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EXECUTIVE  SUMMARY 


The  Ada  community,  including  government,  industry,  and  academic  personnel, 
needs  the  capability  to  assess  APSEs  (Ada  Programming  Support  Environments)  and  their 
components,  and  to  determine  their  conformance  to  applicable  standards  (e.g.,  DoD- 
STD-1838,  the  CAIS  standard).  The  technology  required  to  fully  satisfy  this  need  is  extensive 
and  largely  unavailable;  it  cannot  be  acquired  by  a  single  government-sponsored,  professional 
society-sponsored,  or  private  effort.  The  purpose  of  the  APSE  Evaluation  and  Validation 
(E&V)  task  is  to  provide  a  focal  point  for  addressing  the  need  by: 

(1)  Identifying  and  defining  specific  technology  requirements, 

(2)  .  Developing  selected  elements  of  this  technology, 

(3)  Encouraging  others  to  develop  additional  elements,  and 

(4)  Collecting  information  describing  elements  which  already  exist. 

This  information  will  be  made  available  to  DoD  components,  other  government  agencies, 
industry  and  academia. 

The  purpose  of  the  E&V  Reference  Manual  (this  document)  is  to  provide  informa¬ 
tion  that  will  help  users  to: 


it 

u 
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(1)  Gain  an  overall  understanding  of  APSEs  and  approaches  to  their 
assessment, 

(2)  Find  useful  reference  information  (e.g.,  definitions)  about  specific 
elements  and  relationships  between  elements,  and 

(3)  Find  criteria  and  metrics  for  assessing  tools  and  APSEs,  and 
techniques  for  performing  such  assessment. 

The  latter  are  to  be  found  (or  referenced)  in  a  companion  document  called  the  E&V 
Guidebook. 


Chapters  1  through  3  provide  a  general  introduction  to  the  document  and  to  the  ... 
issue  of  assessing  APSEs  as  a  whole. 
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Chapter  4  and  later  chapters  are  “formal  chapters”  built  around  a  standard  for¬ 
mat  and  formal  grammar.  Each  of  the  formal  chapters  corresponds  to  one  index  of  an  overall 
E&V  Classification  Schema.  The  schema  adopts  a  relational  model  of  the  subject  and  proc¬ 
ess  of  E&V.  This  model  will  allow  the  user  to  arrive  at  E&V  techniques  through  many  different 
paths,  and  provides  a  means  to  extract  useful  information  along  the  way. 

Yearly  updates  and  extensions  to  this  manual  are  planned.  Therefore,  comments 
and  suggestions  are  welcome.  Please  send  comments  electronically  (preferred)  to 
szymansk@ajpo.sei.cmu.edu  or  by  regular  mail  to  Mr.  Raymond  Szymanski,  AFWAL/AAAF, 
Wright  Patterson  AFB,  OH  45433-6543. 
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1.  INTRODUCTION 

1 .1  PURPOSE  OF  THE  MANUAL 

This  document  is  a  product  of  the  Ada  Programming  Support  Environment 
(APSE)  Evaluation  and  Validation  (E&V)  Task  sponsored  by  the  Ada  Joint  Program  Office,  it 
is  one  of  a  pair  of  companion  documents  known  as  the  E&V  Reference  System,  consisting  of: 

•  E&V  Reference  Manual 

•  E&V  Guidebook. 

The  purpose  of  the  Reference  Manual  is  to  provide  a  collection  of  information  to  support  a  va¬ 
riety  of  users.  The  collection  is  organized  in  accordance  with  a  Classification  Schema  de¬ 
scribed  in  Chapter  2.  It  should  help  users  to: 

•  Gain  an  overall  understanding  of  APSEs  and  approaches  to  the 
assessment  of  APSE  performance,  quality  and  conformance  to 
applicable  standards. 

•  Find  useful  reference  information,  such  as  definitions  of  specific 
elements  of  the  Classification  Schema,  and  relationships  be¬ 
tween  elements. 

•  Find  criteria  and  metrics  for  assessing  specific  components, 

combinations  of  components  and  “whole  APSEs,”  and  locate 
relevant  E&V  techniques. 

The  Reference  Manual  includes  many  "pointers"  to  sections  in  the  Guidebook  and  other 
documents  which  describe  E&V  techniques.  Figure  1.1-1  illustrates  the  relationship  between 
the  Reference  Manual  and  the  Guidebook.  Figures  1.1-2  and  1.1-3  illustrate  the  types  of  infor¬ 
mation  to  be  extracted  from  each  document.  Chapter  2  provides  a  more  detailed  description 
of  the  structure  and  uses  of  the  Reference  System. 
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Users  May  Consult 

the  Reference  or 

Manual  to  Extract: 


directly  Consult 
the  Guidebook.. 


(1)  Useful 

Information 
Directly  from  \ 
the  Manual 


Pointers  to 
Sections  in 
the  Guidebook... 


...Which  Provides  Information  About 
E&V  Tools  and  Techniques 


Figure  1.1-1  Uses  of  the  Reference  System 


_ E&V  Reference  Manual,  Version  1.1 _ 

Users  consult  Indexes  in  the  Reference  Manual  c-oo92Sb 


EXAMPLES  of  Types  of  Information  Extracted: 

•  Organization  of  Tha  Indexes 

Hierarchical  Structure  and  a  Numbering  System 
(e.g.,  The  Function  Index  .  .  .  see  Fig.  2.1-2) 

e  Names  and  Descriptions  of  Specific  Elements  within  the  Indexes 
(e.g.,  The  Compilation  Function  .  .  .  see  Figs.  2.1-2  and  2.1-3) 
e  Cross  References  Between  Elements  of  the  Indexee 

(e.g.,  The  Function  “Compilation”  is  Cross  Referenced  to 
Activities  and  Tools  .  .  .  see  Fig.  2.4-1) 

Figure  1.1-2  Examples  of  Reference  Manual  Information 


EXAMPLES  of  Types  of  Information  Extracted: 

e Description  of  Evaluation  (E)  Tools  and  Their  Uses  (e.g.,  see  Section  2.4) 
e  Description  of  Vsiidation  (V)  Suites 

a  References  to  Other  Documents  Describing  E  or  V  Techniques 
e  Synopses  of  Other  Documents  Thet  Cover  Multiple  Techniques 

Figure  1.1-3  Examples  of  Guidebook  Information 
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1.2  USERS  OF  THE  MANUAL 

Classes  of  people  who  are  expected  to  be  users  of  this  manual  are  listed  in  Ta¬ 
ble  1.2-1.  They  are  described  in  terms  of  their  relationships  to  deliverable  software,  tools, 
APSEs,  and  APSE  E&V  technology.  They  may  be  associated  with  Government,  industry,  or 
academia.  The  table  was  adapted  from  material  in  the  report  of  the  E&V  Workshop  [@  E&V 
Workshop  1984].* 

While  the  manual  is  designed  to  be  of  use  to  people  of  all  the  classes  listed,  the 
primary,  users  are  expected  to  be  the  APSE/tool  users  and  the  E&V  technology  users.  These 
are  people  with  highly  technical  backgrounds  and  technical/managerial  interests  in  evaluating 
and  selecting  tools  and  APSEs.  This  expectation  has  strongly  influenced  the  structure  and 
style  of  the  manual.  The  primary  users  are  likely  to  consult  both  the  Reference  Manual  and 
the  Guidebook.  Many  of  the  other  classes  of  users  listed  in  Table  1.2-1  are  likely  to  consult 
the  Reference  Manual  only. 


Table  1.2-1  Reference  Manual  Users 


CLASS 

DESCRIPTION 

Software  Acquisition 
Personnel 

Government  program  officers  and  commercial 
program  managers  who  let  contracts  for 
software  development 

APSE/Tool  Users 

Project  managers,  librarians,  system 
engineers,  software  engineers 

APSE/Tool  Builders 

Environment/tool  designers,  implementers, 
managers,  marketing  personnel 

E&V  Technology  Users 

Government,  commercial  and  university 
personnel  applying  E&V  technology 

E&V  Technology 

Builders 

Anyone  developing  E&V  assessment  techniques 
(E&V  Task  contractors,  etc) 

Investors 

Anyone  funding  development  or  use  of  E&V 
technology  (Congress,  AJPO,  corporations,  etc.) 

*The  format  used  for  references  is  associated  with  the  “formal  grammar”  used 
beginning  with  Chapter  4.  See  further  explanations  in  Appendix  C. 
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1.3  THE  NEED  FOR  E&V  TECHNOLOGY 

Technology  for  the  assessment  of  APSEs  and  APSE  components  (tools)  is 
needed  because  of  the  difficulty  in  assessing  APSEs  and  because  of  the  importance  of  the  de¬ 
cisions  made  based  on  these  assessments.  The  importance  of  an  APSE  selection  is  evident 
when  one  considers  the  large,  critic  ',  Ada-based  systems  to  be  developed  in  the  coming 
years.  The  effectiveness,  reliability,  and  cost  of  these  systems  will  be  strongly  influenced  by 
the  environments  used  to  develop  and  maintain  them.  From  the  point  of  view  of  a  software  de¬ 
veloping  organization,  the  decision  to  select  an  APSE  can  be  an  important  investment  decision 
with  long-lasting  influence  on  a  number  of  projects  and  the  organization’s  methods  of  opera¬ 
tion,  training,  and  competitiveness.  From  the  point  of  view  of  a  software  maintenance  organi¬ 
zation,  the  environment  used  will  strongly  influence  the  organization’s  effectiveness,  as  well  as 
the  cost  of  its  operations  and  training. 

The  difficulty  of  assessing  APSEs  and  tools  exists  for  several  reasons.  First,  an 
APSE  represents  very  complex  technology  with  many  elements,  which  can  be  assessed  indi¬ 
vidually  or  in  combination.  Second  there  is  a  confusing  diversity  of  choice  with  respect  to  indi¬ 
vidual  tools,  tool  sets,  or  “whole  APSEs”;  and  there  are  a  number  of  ways  of  viewing  APSEs; 
see  Chapter  3.  Third,  the  state  of  the  art  of  APSE  architecture  and  of  some  categories  of  tools 
(e.g.,  graphic  design  tools)  is  undergoing  rapid  change.  Finally,  there  is  a  lack  of  historical 
data  relevant  to  APSEs.  partly  because  of  the  general  pace  of  technological  change  and  partly 
because  we  are  dealing  with  Ada.  a  relatively  new  implementation  language.  E&V  technology 
provides  methods  and  techniques  to  overcome  these  difficulties  and  provides  a  basis  for  deter¬ 
mining  performance  and  other  attributes  of  APSEs. 

In  addition  to  the  need  for  assessment  technology  itself,  there  is  a  need  for  infor¬ 
mation  about  this  technology.  Potential  buyers  and  users  of  APSEs  and  tools  need  a  frame¬ 
work  for  understanding  APSEs  and  their  assessment,  as  well  as  information  about  specific 
assessment  techniques.  Similarly,  vendors  of  tools  and  APSEs  need  to  be  aware  of  the  defi¬ 
ciencies  of  current  products,  as  well  as  the  criteria  to  be  used  in  the  assessment  of  future 
products.  Such  awareness  on  both  sides,  expressed  in  a  common  terminology,  should  speed 
up  the  evolution  of  better  software  development  environments. 
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1.4  BACKGROUND 


In  June  1983  the  Ada  Joint  Program  Office  (AJPO)  proposed  the  formation  of  the 
E&V  Task  and  a  tri-service  E&V  Team,  with  the  Air  Force  designated  as  lead  service.  In  Oc¬ 
tober  1983  the  Air  Force  officially  accepted  responsibility  as  lead  service  and  designated  the 
Air  Force  Wright  Aeronautical  Laboratories  (AFWAL)  at  Wright  Patterson  Air  Force  Base  as 
lead  organization,  in  April  1984  an  E&V  Workshop  was  held  at  Airlie,  Virginia.  The  purpose  of 
the  workshop  was  to  solicit  participation  of  industry  representatives  in  the  E&V  Task.  Many  of 
the  participants  in  the  workshop  have  chosen  to  remain  involved  as  Distinguished  Reviewers, 
and  additional  industry  participants  have  subsequently  become  involved  in  E&V  Team  activi¬ 
ties. 


The  E&V  Team  publishes  an  annual  public  report.  The  following  paragraph  is 
quoted  from  the  1987  version  [@E&V  Report  1987]  of  the  report: 


“The  Ada  community,  including  government,  industry,  and  academic  personnel, 
needs  the  capability  to  assess  APSEs  (Ada  Programming  Support  Environ¬ 
ments)  and  components,  and  to  determine  their  conformance  to  applicable 
standards  (e.g.f  DoD-STD-1838,  the  CAIS  standard).  The  technology  required 
to  fully  satisfy  this  need  is  extensive  and  largely  unavailable;  it  cannot  be  ac¬ 
quired  by  a  single  government-sponsored,  professional  society-sponsored,  or 
private  effort.  The  purpose  of  the  APSE  Evaluation  and  Validation  (E&V)  task 
is  to  provide  a  focal  point  for  addressing  the  need  by  (1)  identifying  and  defin¬ 
ing  specific  technology  requirements,  (2)  developing  selected  elements  of  the 
required  technology,  (3)  encouraging  others  to  develop  some  elements,  and  (4) 
collecting  information  describing  existing  elements.  This  information  will  be 
made  available  to  DoD  components,  other  government  agencies,  industry,  and 
academia.” 


The  team  public  reports  contain  much  additional  information  for  the  interested 
reader.  Three  competitive  contracts  have  been  awarded  under  the  E&V  task.  These  are: 


•  Technical  Support  contract  -  awarded  June  1985 

•  Ada  Compiler  Evaluation  Capability  (ACEC)  contract  -  awarded 
February  1987 

•  CAIS  Implementation  Validation  Capability  (CIVC)  contract  - 
awarded  May  1987. 
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The  major  purpose  of  the  first  of  these  contracts  is  to  create  and  update  ele¬ 
ments  of  the  E&V  Reference  System,  including  this  manual.  The  purpose  of  the  second  and 
third  contracts  is  to  create  two  specific  elements  (ACEC  and  CIVC)  of  the  needed  E&V  tech¬ 
nology. 

1 .5  ORGANIZATION  OF  THE  MANUAL 

Chanter  2  discusses  the  structure  of  the  E&V  Reference  System  (Reference  Man¬ 
ual  plus  Guidebook)  and  the  Classification  Schema  upon  which  that  structure  is  based.  Spe¬ 
cific  directions  as  to  how  to  use  the  manual  are  also  included. 

Chapter  3  provides  a  general  discussion  of  “whole  APSE”  issues,  in  which  an 
APSE  is  viewed  as  more  than  the  sum  of  its  parts.  Key  whole-APSE  attributes  are  discussed 
along  with  general  approaches  to  whole-APSE  assessment. 

Chanter  4  and  subsequent  chanters  are  “formal  chapters”  built  around  a  stan¬ 
dard  format  and  formal  grammar.  Each  of  the  formal  chapters  corresponds  to  one  index  of 
the  Classification  Schema.  The  structure  and  use  of  these  chapters  are  the  focus  of  the  ma¬ 
terial  found  in  Chapter  2. 

The  appendices  include  a  description  of  the  formal  grammar,  a  glossary  of  acro¬ 
nyms  and  abbreviations,  a  document  citation  list,  and  a  composite  index. 
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2.  USE  OF  THE  REFERENCE  SYSTEM 


This  chapter  provides  a  step-by-step  explanation  of  how  to  use  the  E&V  Refer¬ 
ence  System  and  the  Classification  Schema  upon  which  the  system  is  based.  Section  2.1  de¬ 
scribes  the  organization  of  the  material.  Sections  2.2,  2.3,  and  2.4  contain  illustrations  of  uses 
of  the  system,  presented  in  increasing  levels  of  sophistication.  Section  2.5  provides  a  global, 
conceptual  view  of  the  system  framework.  User  A  (Section  2.2)  consults  an  index  of  the  Ref¬ 
erence  Manual  to  find  the  description  of  a  term  that  is  an  element  of  that  index.  User  B  (Sec¬ 
tion  2.3)  consults  an  index  to  find  an  element  and  several  cross  references  to  related 
elements  in  another  index.  User  C  (Section  2.4)  consults  a  combination  of  indexes  to  find  ref¬ 
erences  to  sections  in  the  Guidebook,  which  contain  explanations  of  relevant  Evaluation  or 
Validation  techniques.  Brief  definitions  of  several  key  words  and  expressions  follow: 

E&V  —  Evaluation  and  Validation 

Evaluation  —  Assessing  performance  and  quality 

Validation  —  Assessing  conformance  to  a  Standard 

E&V  Reference  System  —  Two  documents: 

the  E&V  Reference  Manual  and 
the  E&V  Guidebook 

E&V  Classification  Schema  —  A  set  of  indexes  that  provide  a 

framework  for  the  E&V  Reference 
Manual. 

The  schema  was  initially  described  in  an  earlier  “E&V  Classification  Schema  Report”  [@E&V 
Classification],  which  was  used  as  the  starting  point  for  the  schema  defined  in  this  manual. 
Subsequent  changes  in  the  schema  will  be  updated  in  future  versions  of  this  manual:  the 
schema  report  will  not  be  updated. 

2.1  SYSTEM  ORGANIZATION 

The  entire  reference  system  can  be  viewed  as  a  structure  of  multiple  indexes. 
For  example,  there  is  a  function  index  and  a  life  cycle  activity  index,  among  others.  The 
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structure  is  analogous  to  the  card  catalog  system  in  a  public  library,  with  its  author  index,  title 
index,  and  subject  index.  To  use  the  card  catalog  in  the  library,  you  must  first  locate  the  card 
that  corresponds  to  the  author,  title,  or  subject  in  which  you  are  interested.  Similarly,  to  use 
the  Reference  Manual,  you  first  find  the  element(s)  in  which  you  are  interested.  The  way  to 
do  this  is  to  begin  by  looking  at  the  Table  of  Contents  or  the  Index  at  the  back  of  the  manual 
to  locate  the  element(s)  in  the  “formal  chapters.”  The  names  of  the  indexes  that  are  formal 
chapters  within  the  Reference  Manual  are: 

•  Life  Cycle  Activities  (Chapter  4) 

•  APSE  Tool  Categories  (Chapter  5) 

•  Attributes  (Chapter  6) 

•  Functions  (Chapter  7). 

Figures  2.1-1,  2.1-2,  and  2.1-3  provide  a  pictorial  view  of  the  organization  of  the 
Reference  Manual,  particularly  the  structure  of  the  reference  material  contained  in  the  “middle 
section.”  The  chapters  of  this  middle  section  ara  organized  in  a  consistent,  formal  manner, 
using  a  formal  grammar  (described  in  Appendix  C).  A  typical  chapter  corresponds  to  one  in¬ 
dex  of  the  schema.  A  typical  index  is  organized  as  a  hierarchical  structure  of  elements.  For 
every  element  there  is  a  “text  frame”  that  has  (in  general)  three  parts  as  shown  in  Fig.  2.1-3. 
The  text  frames  are  built  using  the  formal  grammar.  (Details  of  the  formal  grammar  need  not 
concern  the  user.  It  was  employed  because  of  the  possibility  of  a  future  on-line,  electronic 
version  of  the  system,  supported  by  advanced  updating  and  information  retrieval  techniques.) 
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The  E&V  Reference  Manual  o-oo927b 

Has  Three  Parts,  As  Shown 


Typically  a  chapter  in  the  "Reference  Material”  section  corresponds,  to  one 
index  of  the  Classification  Schema.  These  chapters  are  organized  in  a  consistent, 
formal  manner. 


Figure  2.1-2  Reference  Material  Indexes 
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Cross  References:  Identify  significant  relationships  with  other  elements 

in  the  same  index  or  other  indexes  of  the  Reference 
Manual. 


Guidebook  References:  Identify  relationships  involved  in  the  statements  of 

E&V  objectives  and  point  to  sections  in  the  Guide¬ 
book  where  information  about  how  to  satisfy 
objectives  may  be  found. 


Figure  2.1-3  Text  Frames 
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2.2  DESCRIPTION:  DIRECT  REFERENCE 


Our  first  example  of  the  use  of  the  Reference  Manual  is  "User  A”  who  consults 
the  Attributes  Index  to  find  the  description  of  the  term  “Storage  Effectiveness.”  Figure  2.2-1  is 
a  copy  of  Section  (or  Text  Frame)  6.4.31 .  User  A  may  find  this  frame,  for  example,  by  brows¬ 
ing  through  the  Table  of  Contents  or  by  looking  up  the  term  “Storage  Effectiveness”  in  the 
main  Index  at  the  back  of  the  manual.  Note  that  the  text  frame  contains  Cross  References 
and  Guidebook  References  as  well,  but  this  need  not  concern  User  A,  who  simply  seeks  a  de¬ 
scription.  The  User  A  scenario  is  pictorially  represented  in  Fig.  2.2-2,  where  the  boxes  are 


analogous  to  cabinets  in  a  library  card  catalog  system. 


6.4.31  Storage  Effectiveness 

Description: 

Those  characteristics  of  the  software  which  provide  for  minimum  utilization  of  storage  re¬ 
sources  in  performing  functions.  [@RADC  1985]  The  choice  between  alternative  source  code 
constructions  based  on  those  taking  the  minimum  number  of  words  of  object  code  or  in  which 

the  information-packing  ...  is  high.  [@DACS  1979] 

Cross  Ref erencflai 

Software  Quality  Factors: 

[Efficiency 

Beneficial  Quality  Factors: 

Adverse  Quality  Factors: 

6.1.1] 

[Maintainability 

6.2.2, 

Verifiability,  Testability 

6.2.3, 

Transportability 

6.3.4] 

Guidebook-  References: 

[Preliminary  Design 

7.1. 6.4, 

@GB:  SEI  Design  Support  Experiment 

9.1; 

Detailed  Design 

7. 1.6.5, 

@GB:  SEI  Design  Support  Experiment 

9.1; 

'Compilation 

7. 1.6. 7, 

<®GB:  IDA  Benchmarks 

5.2; 

'Compilation 

7. 1.6. 7, 

@GB:  ACEC 

5.3; 

'Compilation 

7.1. 6.7, 

@GB:  PIWG  Benchmark  Tests 

5.4; 

'Compilation 

7.1. 6. 7, 

@GB:  ARTEWQ  Catalogue  of  Ada  Runtime 

Implementation  Dependencies 

5.10] 

'NOTE:  Normally,  the  concern  of  evaluation  is  the  quality  of  a  tool. 

However,  this  evaluation 

techniaue  focuses  on  the  oroducts  of  the  tool  rather  than  the  tool  itself. 

Figure  2.2-1  Sample  Attributes  Index  Frame 
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USER  A 


G-06646 

09/19/88 


2.3  CROSS  REFERENCE 

Our  second  example  of  use  of  the  Reference  Manual  is  "User  B”  who  would  like 
to  know  the  names  and  descriptions  of  functions  associated  with  a  particular  group  of  life  cy¬ 
cle  activities.  User  B  first  consults  the  Life  Cycle  Activities  Index,  Chapter  4,  which  divides  the 
life  cycle  in  accordance  with  the  DoD-STD-2167A  model  [@DoD-STD-2167A]  for  project  de¬ 
velopment.  Chapter  4  relates  the  life  cycle  activity  groups  to  the  functions  typically  found 
within  an  APSE  and  the  products  that  are  produced  in  each  portion  of  the  life  cycle.  A  sample 
Life  Cycle  Activities  Index  Frame  is  shown  in  Fig.  2.3-1 .  This  frame  represents  the  sixth  activ¬ 
ity  group  [4.6  Coding  and  CSU  Testing]  and  the  second  class  of  activities  [4.6.2  Software  En¬ 
gineering]  covered  in  each  activity  group.  The  Software  Development  File  is  the  product  for 
this  group  of  activities.  User  B  finds  13  functions  that  might  be  performed  during  this  activity 
listed  in  the  text  frame.  In  each  case  another  text  frame  is  cross  referenced  in  Chapter  7.  the 
Functions  Index,  where  more  information  about  this  function  is  found.  The  User  B  scenario  is 
pictorially  represented  in  Fig.  2.3-2. 
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4.6.2  Transformation 

Cross  References: 

Products: 

[Software  Development  File 

(SDF)] 

Functions: 

[Predefined  and  User-Defined  Forms 

7.1. 2.3, 

Assembling 

7.1. 6.6, 

Compilation 

7.1. 6.7, 

Conversion 

7. 1.6.8. 

Macro  Expansion 

7. 1.6.9, 

Structure  Preprocessing 

7.1.6.10, 

Body  Stub  Generation 

7.1.6.11, 

Preamble  Generation 

7.1.6.12, 

Linking/Loading 

7.1.6.13, 

Interpretation 

7.1.6.14, 

Program  Generation 

7.1. 7.3, 

Program  Library  Management 

7.2.1. 7. 

Runtime  Environment 

7.2.3.5J 

Figure  2.3-1  Sample  Life  Cycle  Activity  Index  Frame 


2.4  GUIDEBOOK  REFERENCE 

Our  third  example  of  use  of  the  Reference  Manual  is  "User  C”  who  would  like  to 
look  up  a  function,  learn  what  attributes  are  associated  with  it,  and  find  evaluation  techniques 
relevant  to  particular  function-attribute  pairs.  User  C  first  consults  the  Function  Index,  Chap¬ 
ter?.  A  sample  Function  Index  Frame  [7.1. 6.7  Compilation]  is  shown  in  Fig.  2.4-1.  Func¬ 
tions  are  related  to  life  cycle  activities  and  tools.  In  the  example,  the  function,  Compilation,  is 
related  to  five  life  cycle  activity  groups:  Coding  and  CSU  Testing,  CSC  Integration  and  Test¬ 
ing,  CSCI  Testing,  System  Integration  and  Testing,  and  Operational  Testing  and  Evaluation. 
The  user  of  the  manual  can  refer  to  Chapter  4  to  find  more  information  about  these  activity 
groups.  The  Compiler  is  the  tool  that  performs  the  function  Compilation.  Information  about 
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USER  B  G-06647 


Figure  2.3-2  Example  of  Cross  Reference:  User  B 


compilers  can  be  found  in  Chapter  5.  User  C  is  interested  in  evaluation  of  the  compilation 
function  with  respect  to  various  attributes.  For  example,  the  function-attribute  pair 
Compilation-Processing  Effectiveness  is  represented  by  the  items  under  Guidebook  Refer¬ 
ences  in  this  text  frame.  This  pair  points  to  sections  in  the  Guidebook  called  IDA  Benchmarks 
and  ACEC,  among  others,  which  provide  additional  information  on  these  two  E&V  techniques. 
A  user  of  the  Reference  Manual  can  find  references  to  these  two  techniques  in  the  Attribute  In¬ 
dex  or  the  Function  Index.  The  User  C  scenario  is  pictorially  represented  in  Fig.  2.4-2. 
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7. 1.6.7  Compilation 

Description; 

Translating  a  computer  program  expressed  in  a  procedural  or  problem-oriented  language 
into  object  code.  [@Kean  1985] 


Life  Cycle  Activities: 

[Coding  And  CSU  Testing 
CSC  Integration  And  Testing 
CSCI  Testing 

System  Integration  And  Testing 
Operational  Testing  And  Evaluation 

Tools: 

[Compiler 


4.6.2, 

4.7.2, 

4.8.2, 

4.9.2, 
4.10.2] 

5.3.3] 


*  [Anomaly  Management  6.4.2, 


Capacity  6.4.6, 

Capacity  6.4.6, 

Completeness  6.4.9, 

Power  6.4.21 , 

*  Processing  Effectiveness  6.4.22, 

*  Processing  Effectiveness  6.4.22, 

*  Processing  Effectiveness  6.4.22, 

*  Processing  Effectiveness  6.4.22, 

*  Processing  Effectiveness  6.4.22, 


*  Retargetability  6.4.27, 


*  Storage  Effectiveness  6.4.31, 

*  Storage  Effectiveness  6.4.31, 

*  Storage  Effectiveness  6.4.31, 

*  Storage  Effectiveness  6.4.31, 


ARTEWG  Catalogue  of  Ada 
Runtime  Implementation 
Dependencies 
IDA  Benchmarks 
MITRE  Benchmark 
Generator  Tool 
ACVC 

Compilation  Checklist 
IDA  Benchmarks 
ACEC 

PIWG  Benchmark  Tests 
University  of  Michigan 
Benchmark  Tests 
ARTEWG  Catalogue  of  Ada 
Runtime  Implementation 
Dependencies 
ARTEWG  Catalogue  of  Ada 
Runtime  Implementation 
Dependencies 
IDA  Benchmarks 
ACEC 

PIWG  Benchmark  Tests 
ARTEWG  Catalogue  of  Ada 
Runtime  Implementation 
Dependencies 


NOTE:  Normally,  the  concern  of  evaluation  is  the  quality  of  a  tool.  However,  this  evaluation 
technique  focuses  on  the  product  of  the  tool  rather  than  the  tool  itself. 


Figure  2.4-1  Sample  Functions  Index  Frame 
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USER  C 


G-06648 

9/26/88 


2.5  REFERENCE  FRAMEWORK 

Figure  2.5-1  depicts  the  Classification  Schema  as  an  internal  framework  for  the 
Reference  Manual.  The  schema  provides  paths,  within  and  between  indexes,  that  users  can 
follow  to  extract  information  directly  or  to  find  sections  in  the  Guidebook  that  describe  ele¬ 
ments  of  E&V  technology.  The  figure  also  indicates  the  direct  relationships,  that  is,  types  of 
cross  references  and  Guidebook  references  featured  in  the  Reference  Manual. 

The  Function  Index  is  seen  to  be  directly  related  to  all  of  the  other  indexes. 
Thus,  it  is  drawn  in  an  “anchoring"  position  in  the  diagram.  Attributes,  also,  play  a  key  role  in 
the  overall  E&V  process,  in  two  ways.  First,  many  assessment  objectives  are  defined  in  terms 
of  function-attribute  pairs,  such  as  compilation-efficiency  and  editing-power.  Second,  other 
attributes  (those  not  “pairing-up"  with  functions)  represent  factors  or  criteria  by  which  APSEs 
or  APSE  components  are  assessed,  independent  of  the  functions  performed.  Examples  of  the 
latter  are:  maintainability,  interoperability,  maturity,  and  cost. 
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G-05097A 

09/27/88 


USERS 


AND  TECHNIQUES 
DESCRIBED  IN  THE 
GUIDEBOOK 


Figure  2.5-1  The  Schema  as  Framework  for  the  Reference  Manual 


Besides  the  direct  relationships  indicated  in  Fig.  2.5-1 ,  ir  iirect  relationships  can 
also  be  useful  and  are  constructed  by  combining  two  or  more  direct  relationships.  For  exam¬ 
ple,  since  activities  and  functions  are  related  and  functions  and  tools  are  related,  it  is  possible 
to  determine  the  relationships  between  activities  and  tools.  It  should  be  noted  that  not  all  such 
constructions  will  be  useful.  Such  a  useless  relationship  might  be  life  cycle  activities  and  at¬ 
tributes  related  via  function. 


The  conceptual  structure  pictured  in  Fig.  2.5-1  has  an  open-ended  quality  in  sev¬ 
eral  ways.  First,  there  is  no  fixed  number  of  indexes;  more  can  be  added  as  desired.  As 
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each  new  index  is  added  a  new  section  of  the  Reference  Manual  can  be  added,  providing  ap¬ 
propriate  definitions  and  cross-references  to  other  indexes.  The  new  index  would  be  repre¬ 
sented  by  an  additional  block  along  the  “main  diagonal.”  Second,  each  individual  index  may 
have  elements  added  to  it  as  new  understanding  of  the  various  aspects  of  APSEs  and  the 
E&V  process  is  gained.  The  process  of  modifying  the  structure  of  an  individual  index  in  this 
way  would  take  place  within  the  boxes.  Finally,  the  off-diagonal  space,  above  and  below  the 
main  diagonal,  represents  the  notion  that  any  two-way  relationship  may  be  included  in  the  sys¬ 
tem.  Thus,  any  section  may  be  referenced  by  other  sections,  and  any  section  may  reference 
other  sections.  Also,  any  two  indexes  may  be  involved  in  a  relationship  of  the  type  that  de¬ 
fines  an  E&V  objective  or  points  to  an  E&V  technique  in  the  Guidebook.  Although  many  of  the 
potential  combinations  are  not  expected  to  be  useful  or  relevant  to  E&V  purposes,  the  struc¬ 
ture  permits  the  consideration  of  ail  possibilities. 

The  procedures  described  in  the  Guidebook  are  organized  into  chapters,  as  indi¬ 
cated  in  Fig.  2.5-2.  Chapters  5  and  beyond  are  “formal  chapters”  built  on  a  standard  format. 
Most  of  the  pointers  from  the  Reference  Manual  to  the  Guidebook  refer  to  specific  instances  of 
evaluation  (E)  or  validation  (V)  techniques  described  in  these  formal  chapters.  There  are  also 
some  “backward  references”  from  these  chapters  to  Chapter  4,  “Synopses”  [@GB  4]  or  to 
the  Reference  Manual. 

Evaluation  techniques  can  be  characterized  as  based  on  subjective  judgments  or 
objective  procedures,  designed  to  assess  performance  and  quality.  Validation  techniques 
can  be  characterized  as  based  on  informal  or  formal  procedures,  designed  to  assess  confor¬ 
mance  to  a  standard.  The  types  of  E&V  techniques  described  in  the  Guidebook  fall  into  the 
following  categories: 

•  Questionnaire 

•  Capabilities  Checklist 

•  Assessment  Checklist 

•  Test  Suite 

•  Automated  Test  Suite 

•  Structured  Experiment. 

Examples  of  all  of  the  above  are  spread  throughout  the  Guidebook.  Chapter  3  of  the  Guide¬ 
book  [@GB  3]  provides  a  discussion  of  how  to  combine  results  from  a  variety  of  techniques  to 
perform  an  integrated  whole-APSE  assessment. 
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G- 11 880 
09/27/88 


E&V  GUIDEBOOK 

1 .  Introduction 

2.  Structure  and  Use  of  the  Guidebook 

3.  Integration  of  E&V  Technology 

4.  Synopses 


5.  Compilation  System  Assessors 

6.  Target  Code  Generation  Aids  and  Analysis 
Assessors 

7.  Test  System  Assessors 

8.  Tool/Host  Interface  Assessors 

9.  Requirements/Design  Support  Assessors 

10.  Configuration  Management  Support  Assessors 

11.  Distributed  System  Development  and  Runtime 
Support  Assessors 

12.  Distributed  APSE  Assessors 

13.  “Whole  APSE"  Assessors 

14.  Adaptation  Assessors 

99.  Other  Assessors 


E&V  TECHNIQUES 


Figure  2.5*2  Organization  of  the  Guidebook 
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3.  WHOLE  APSE  ASSESSMENT  ISSUES 


This  chapter  is  intended  as  a  starting  place  for  those  interested  in  selecting  or 
assessing  an  APSE  considered  as  a  “whole  entity"  that  is  “more  than  the  sum  of  its  parts." 
The  chapter,  therefore,  cannot  be  organized  in  the  index/text  frame  style  of  other  chapters  in 
this  manual,  because  that  style  fits  the  view  of  an  APSE  as  a  collection  of  components  or 
functions  that  can  be  evaluated  one  at  a  time  and  “added  up."  Here  we  take  the  opposite 
view  —  that  the  APSE  is  a  total  system  which  serves  a  project  team  across  an  entire  software 
development  life  cycle  —  and  that  it  should  be  evaluated  in  terms  of  overall  project  goals  and 
team  productivity,  rather  than  in  terms  of  individual  atomic  functions. 

Although  the  technology  of  “whole  APSE  assessment"  is  (at  this  writing)  very 
immature,  there  are  many  helpful  materials  in  the  open  literature.  This  chapter  may  be 
considered  a  guide  to  that  literature.  It  is  organized  to  address  the  following  types  of 
questions: 


What  is  an  APSE? 

How  can  whole  APSEs  be  viewed? 

What  are  the  key  whole  APSE  attributes? 

How  can  whole  APSEs  be  assessed? 

Where  can  relevant  information  be  found? 

The  first  four  questions  are  addressed  in  Sections  3.1  through  3.4,  respectively.  Various 
references  to  other  sections  of  the  E&V  Reference  Manual  (RM),  the  E&V  Guidebook  (GB), 
and  open-literature  sources  are  distributed  throughout  the  chapter. 


3.1  APSE  DEFINITIONS  AND  ALTERNATIVE  NAMES 

The  acronym  APSE  stands  for  “Ada  Programming  Support  Environment."  The 
term  "Programming  Support  Environment”  is  defined  in  the  “IEEE  Standard  Glossary  of 
Software  Engineering  Terminology"  [@IEEE  1983]  as 
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“An  integrated  collection  of  tools  accessed  via  a  single  command  language  to 
provide  programming  support  capabilities  throughout  the  software  life-cycle. 

The  environment  typically  includes  tools  for  design,  editing,  compiling,  loading, 
testing,  configuration  management,  and  project  management.” 

Thus,  one  useful  definition  of  an  APSE  is  the  above  quotation  along  with  the  stipulation  that 
there  be  at  least  one  Ada  compiler  among  the  tools  provided.  A  similar  definition  for  the  term 
IPSE,  which  stands  for  “Integrated  Project  Support  Environment,”  is  given  [@Lehman  and 
Turski  1987]  as 

“An  embodiment  of  software  technology  in  a  collection  of  tools  for  capture, 
representation,  control,  refinement,  transformation,  and  other  manipulation  of 
project  related  information." 

The  second  name  and  definition  are  broader  than  the  first  because  they  refer  to  total 
“project”  support  and  information,  rather  than  “programming”  support  and  programs.  The 
distinction  between  the  two  is  highlighted  not  only  in  the  Lehman  and  Turski  paper,  but  in 
many  others,  including  a  survey  paper  by  Houghton  and  Wallace  [@Houghton  and  Wallace 
1987].  The  latter  uses  the  following  terms  to  characterize  most  existing  environments: 

•  Framing  Environments 

•  Programming  Environments 

•  General  Environments. 

Framing  environments  concentrate  on  the  early  stages  of  the  life  cycle  and  tend  to  be 
methodology-specific.  Programming  environments  concentrate  on  the  latter  part  of  the  life 
cycle  and  are  oriented  toward  programming,  debugging,  and  testing.  General  environments 
contain  basic  tools  that  support  all  phases  of  the  life  cycle  and  tend  to  be  methodology-free. 

Both  of  the  above  definitions  and  discussion  are  limited  in  that  they  are  tied  to 
one,  traditional  view  of  an  APSE.  In  this  view  an  APSE  (or  IPSE)  is  seen  as  a  collection  of 

tools.  The  following  section  presents  additional  views,  any  of  which  may  become  increasingly 

important  to  those  who  wish  to  select  or  evaluate  APSEs  of  the  future. 

Additional  acronyms  found  in  the  literature  on  this  same  general  topic  are  listed 
below,  among  others  mentioned  previously. 
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APSE  -  Ada  Programming  Support  Environment 
IPSE  -  Integrated  Project  Support  Environment 
PSE  -  Programming  Support  Environment 
SDE  -  Software  Development  Environment 
SEE  -  Software  Engineering  Environment 

The  term  SDE,  for  example,  has  been  used  in  the  title  of  several  ACM-sponsored 
conferences,  and  is  the  subject  of  an  IEEE  tutorial  [@IEEE  1981].  All  of  the  above  are 
possible  key  words,  under  which  may  be  found  useful  material  on  whole  APSEs  or  whole 
APSE  E&V,  the  subject  of  this  chapter. 

A  definition  of  an  APSE,  based  on  the  concept  of  a  collection  of  tools,  is  limited  in 
the  sense  that  it  describes  only  a  portion  of  the  total  environment  for  software  development. 
As  discussed  in  the  paper  “The  Ecology  of  Software  Environments"  [@Wasserman  1981a], 
the  total  environment  or  “surroundings”  includes  such  things  as  the  computer  itself,  peripheral 
storage  and  display  devices,  project  management  practices,  organizational  characteristics, 
and  external  constraints  such  as  governmental  directives  and  physical  workspace  factors.  All 
of  the  above  will  influence  the  development  and  application  of  whole  APSE  evaluation 
techniques. 


3.2  VIEWS  OF  AN  APSE 

Various  ways  of  viewing  an  APSE  are  summarized  briefly  below.  The  views  differ 
in  terms  of  both  how  an  APSE  is  seen  —  that  is,  what  sort  of  conceptual  model  does  the 
viewer  hold  —  and  bv  whom  is  the  APSE  viewed  —  for  example,  the  APSE  user  would  typically 
have  a  different  view  than  the  APSE  builder.  The  views  presented  are  not  necessarily 
mutually  exclusive;  an  individual’s  view  of  a  specific  APSE  may  combine  aspects  of  several  of 
the  following  notions. 

3.2.1  APSE  Viewed  as  a  Collection  of  Tools 

This  is  the  traditional  view  of  programming  support  environments,  and  is 
consistent  with  the  standard  definition  quoted  in  the  previous  section.  An  advantage  of  taking 
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this  view  is  that  it  permits  the  viewer  to  characterize  an  APSE  in  terms  of  elements  of  a 
functional  taxonomy,  such  as  that  provided  in  Chapter  7  [Functions  7].  The  characterization 
can  be  stated  in  terms  of  yes/no  answers  to  a  long  list  of  clearly  defined  questions  —  is 
function  x.y.z  provided  or  is  it  not?  A  disadvantage  of  this  view  is  that  it  may  cause  the  viewer 
to  neglect  the  crucial  whole-APSE  issues  that  are  the  subject  of  this  chapter,  and  which  are 
impossible  to  express  in  terms  of  a  composition  of  individual  low-level  functions. 

One  version  of  this  view  is  given  in  the  Stoneman  Report  [@DoD  1980],  which 
pictures  a  multi-layered,  extensible  collection  built  around  an  inner  kernel  (KAPSE)  and  a 
surrounding  minimal  (MAPSE)  layer  of  essential  tools.  This  version  may  have  more  relevance 
for  APSE  builders  than  for  APSE  users,  since  users  need  not  necessarily  be  aware  of  the  layer 
in  which  a  particular  tool  resides. 

3.2.2  APSE  Viewed  as  a  Methodology-Support  System 

In  this  view  an  APSE  is  seen  as  a  system  that  supports  a  particular  development 
methodology  or  a  particular  model  of  the  software  development  process.  It  might  be  strongly 
tied  to  a  standard  set  of  products  and  activities  such  as  those  required  by  the  U.S. 
Department  of  Defense  [@DoD-STD-2167A]  for  mission  critical  software.  It  might  be  based 
upon  a  coherent  software  development  and  maintenance  methodology,  such  as  that 
described  in  “Life-Cycle  Support  in  the  Ada  Environment"  [@McDermid  and  Ripken  1984]. 
The  latter  expresses  the  view  that,  “The  purpose  of  a  tool  is  to  support  a  method  by 
automating  some  aspect  of  the  use  or  application  of  the  method.” 

3.2.3  APSE  Viewed  as  an  Information  Management  System 

In  this  view  primary  emphasis  is  placed  on  how  project  information  is  stored, 
retrieved,  manipulated,  and  maintained  over  the  entire  life  cycle.  A  key  issue  is  the  set  of 
interfaces  between  tools  and  the  project  data  base  [@Houghton  and  Wallc.ee  1987];  the  data 
base  may  be  considered  a  tool  that  is  used  by  other  tools  and  therefore  is  the  interface 
between  them.  Another  key  issue  is  the  control  of  access  to  information,  and  how  this  is 
affected  by  data  structure  models  —  such  as  hierarchical,  relational,  transactional,  etc. 
McDermid  [@McDermid  1985)  envisions  three  generations  of  environments  as  follows.  The 
first  generation  would  be  a  conventional  collection  of  Ada  tools  tied  to  a  Unix-like  environment. 
The  second  generation  would  be  based  on  a  data  base  schema  that  provides  appropriate  life 
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cycle  support,  but  has  a  static,  non-adaptive  structure.  The  third  generation  would  provide  a 
dynamic,  adaptive  information  management  system  that  can  encapsulate  any  underlying  data 
base  schema. 

Another  key  set  of  issues  to  be  addressed  are  those  specific  to  the  Ada 
compilation  process.  A  unique  feature  of  the  Ada  language  is  that  inter-unit  dependencies 
exist  at  compilation  time.  Source  code  units  can  be  compiled  separately,  but  not 
independently.  The  implications  of  this  feature  are  discussed  further  in  Section  3.3.1  under 
Integrity. 

3.2.4  APSE  Viewed  as  a  User-Oriented,  Interactive  System 

This  view  emphasizes  user  interfaces  and  human  factors.  According  to  the 
authors  of  a  survey  paper  [@Houghton  and  Wallace  1987],  “Many  systems  that  measure 
poorly  in  terms  of  human  factors,  including  most  traditional  operating  systems,  have  withstood 
the  test  of  time.  .  .  The  real  users  are  the  systems  programmers  or  gurus. "  (But.  in  the 

future)  “Software  engineering  environments  .  .  .  should  not  require  a  system  expert  as  an 
interface  between  the  software  engineer  and  the  environment.”  A  conference  overview  paper 
[©Henderson  1987]  sees  “a  trend  toward  the  use  of  graphics."  The  desire  for  graphical 
interactive  workstations  is  echoed  in  a  paper  on  personal  development  systems  [@Gutz  et  al 
1981]  with  the  words,  “time  sharing  is  an  idea  whose  time  is  gone.”  An  important  set  of 
issues  concerns  the  different  roles  of  various  users  of  the  system,  different  modes  of  use  for 
each,  and  how  the  system  orients  its  support  to  the  current  user/mode.  An  influential  set  of 
concepts  has  come  from  the  world  of  artificial  intelligence,  such  as  the  “incremental 
enrichment”  style  of  LISP  program  development  [@Barstow  and  Shrobe  1981]. 

3.2.5  APSE  Viewed  as  a  Knowledge-Based  Expert  System 

In  this  view  components  of  an  APSE  (or  conceivably  the  entire  APSE)  are  created 
as  expert  systems  based  on  the  past  experience  of  human  experts  in  specific  domains.  One 
issue  of  IEEE  Expert  [@IEEE  Expert  1986]  contains  a  collection  of  papers  which  address  this 
possibility.  A  paper  in  this  collection  [@Zualkernan  et  al  1986]  outlines  some  steps  to 
determine  the  feasibility  of  this  approach  and  treats  the  specific  area  of  software  testing  as  a 
case  study.  This  view  may  be  considered  a  special,  advanced  case  of  the  preceding  view, 
where  the  style  of  user-interaction  is  that  of  an  “expert  assistant.” 
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3.2.6  APSE  Viewed  as  a  Stable  Framework 

This  view  has  been  advocated  by  C.M.  McKay  of  U.  Houston  Clear  Lake 
[@McKay  1987],  and  is  motivated  by  the  need  to  support  large,  complex,  non-stop, 
distributed,  long-lived  systems  such  as  the  NASA  Space  Station.  The  APSE  is  viewed  as  a 
stabie  framework  to  which  new  or  improved  tools  can  be  added  over  time  without  interfering 
with  or  invalidating  previous  work.  The  framework  is  defined  in  terms  of  standard  phases  and 
deliverables,  in  one  dimension,  and  stable  interface  sets  separating  tools  and  various  classes 
of  objects,  in  the  other  dimension.  Ideally,  the  stability  of  the  framework  and  its  interface  sets 
will  allow  both  the  flight  system  and  its  support  environment  to  evolve  incrementally  as  reliable, 
maintainable  systems. 


3.3  KEY  ATTRIBUTES  OF  WHOLE  APSES 

This  section  provides  a  brief  discussion  of  what  appear  to  be  some  of  the  key 
whole-APSE  attributes.  The  discussion  is  organized  in  accordance  with  the  attribute  taxonomy 
[Attributes  6]  given  in  Chapter  6  of  this  manual,  which  employs  three  top-level  “acquisition 
concern”  categories: 

•  Performance  Attributes  (6.1) 

•  Design  Attributes  (6.2) 

•  Adaptation  Attributes  (6.3). 

Under  each  of  these  is  a  second-level  set  of  “quality  factors",  such  as  Efficiency,  Integrity, 
etc.  These  are  further  decomposed  into  “criteria”,  some  of  which  apply  to  more  than  one 
quality  factor.  All  the  criteria  attributes  are  listed  alphabetically  in  Section  6.4  of  the  Attributes 
Index. 


The  definitions  given  in  Chapter  6  are  very  component-oriented.  For  example, 
the  definition  of  Efficiency  [Efficiency  6.1.1]  includes  the  words,  “The  extent  to  which  a 
component  fulfills  its  purpose  using  a  minimum  of  computing  resources.”  A  whole-APSE 
version  of  the  definition  of  Efficiency  might  read,  “The  extent  to  which  an  APSE  supports  life 
cycle  activities  using  a  minimum  of  development-team  resources.”  This  same  kind  of 
language  modification  is  used  throughout  the  discussion  to  follow. 
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The  selection  of  attributes  highlighted  below  has  been  influenced  by  discussions 
within  the  E&V  Team  and  by  the  following  papers  (some  of  which  make  reference  to  many 
other  papers):  “Toward  Integrated  Software  Development  Environments”  [@Wasserman 
1981b],  “Software  Development  Environment  Issues  as  Related  to  Ada”  [@Notkin  and 
Haberman  1981],  “Essential  Properties  of  IPSEs"  [@Lehman  and  Turski  1987],  and 
“Characteristics  and  Functions  of  Software  Engineering  Environments:  An  Overview” 
[@Houghton  and  Wallace  1987]. 

A  quotation  from  another  paper  by  Wasserman  [@Wasserman  1981  a]  perhaps 
best  sets  the  stage  for  this  discussion  of  key  whole-APSE  attributes.  He  says,  “The  goal  is  to 
create  an  environment  that  not  only  enhances  developer  productivity  but  also  supports  the 
creation  of  superior  products." 

3.3.1  Performance  Attributes 

fEfficiencv  6.1.1]  —  As  stated  above,  efficiency,  in  the  whole-APSE  context,  is 
measured  in  terms  of  resources  used  by  an  entire  team  in  performing  an  entire  group  of 
activities  or  major  “chunk”  of  work  during  development  of  a  software  product.  This  attribute  is 
clearly  related  to  the  overall  project  goal  of  team  productivity. 

[Integrity  6.1.2]  —  This  attribute  deals  with  the  extent  to  which  access  to  the 
software  environment  or  other  data  is  controlled,  especially  for  the  purposes  of  monitoring 
status,  preserving  integrity  of  different  versions,  and  controlling  changes.  It  is  an  attribute  of 
an  important  set  of  management  functions  that  greatly  influence  team  productivity  and  product 
quality. 


A  critical  aspect  of  this  attribute  concerns  the  treatment  of  the  unique  Ada  feature 
(discussed  in  Section  3.2.3)  associated  with  inter-unit  dependencies.  In  managing  Ada 
program  library  units  strict  rules  apply  to  the  order  of  compilation.  As  a  project  is  developed, 
obsolete  units  need  to  be  recompiled,  compilation  order  changes,  and  Ada  closure  sets  may 
be  redefined.  Therefore,  the  APSE’s  management  of  Ada  source  and  object  modules  should 
be  assessed  with  these  special  requirements  in  mind. 
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and 

fUsabilitv/Maturitv  6.1.5/6.4.18]  —  These  are  the  aspects  of  usability  that  naturally 
concern  managers  and  controllers  of  facility  investment  resources.  Anomaly  management 
affects  the  probability  that  an  APSE  will  be  functionally  ready  at  some  specified  point  in  time. 
It  must  operate,  of  course,  on  available  hardware.  Absence  of  such  availability  in  a  timely 
way  would  naturally  be  disastrous. 

Cost  includes  the  cost  of  purchase  or  lease,  installation,  user  assistance,  and 
maintenance.  Its  importance  is  self-evident. 

Maturity  is  the  extent  to  which  an  APSE  has  been  used  in  the  development  of 
deliverable  software  by  typical  users  and  to  which  the  feedback  from  that  use  has  been 
reflected  in  improvements. 

rUsabilitv/Ooerabilitv  6.1.5/6.4.20] 
and 

rUsabilitv/Trainina  6.1.5/6.4.36]  —  These  represent  the  human-factors  aspects 
of  usability  such  as  ease-of-use,  ease-of-learning,  on-line  help  features,  and  consistency  of 
interfaces.  This  set  of  attributes  will  have  a  major  influence  not  only  on  long-term  productivity, 
but  on  the  early  acceptance  of  an  APSE  by  individuals  and  the  team  as  a  whole.  The 
resulting  impact  on  motivation  and  team  spirit  can  be  crucial. 

3.3.2  Design  Attributes 

fCorrectness/Completeness  6.2. 1/6. 4.9]  —  This  is  the  extent  to  which  an  APSE 
supports  the  complete  set  of  operations  necessary  to  perform  its  intended  function,  which  is  to 
provide  full  support  to  a  development  team  across  an  entire  product  life  cycle.  This  attribute 
could  be  interpreted  in  a  minimal  way,  listing  only  truly  essential  (MAPSE)  functions  necessary 
for  a  particular  project.  Alternatively,  it  could  be  interpreted  as  a  list  of  functions  desired  to 
provide  the  capability  to  produce  high-quality  products. 

rMaintainabilitv/Self-Descriotiveness  6.2.2/6.4.28]  —  In  the  whole-APSE  context 
the  aspect  of  this  attribute  to  be  emphasized  concerns  the  underlying  data  base  or  schema 
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used  to  store  and  retrieve  project  data.  In  one  of  the  papers  cited  above  [(©Lehman  and 
Turski  1987]  this  attribute  was  called  data-structuredness.  The  Ada-Europe  document 
“Selecting  an  Ada  Environment”  [(©Lyons  and  Nissen  1986]  says,  “It  is  now  generally 
recognized  that  the  totality  of  information  that  a  project  must  store,  consists  not  only  of 
individual  entities  containing  data  but  also  of  the  relationships  between  them,  such  as  the  facts 
that  a  particular  object  file  has  been  compiled  from  a  particular  source  file,  that  a  source  file 
implements  a  particular  specification,  or  that  an  error  report  relates  to  a  particular  release  of  a 
system.” 


fVerifiabilitv.  Testability  6.2.3]  —  This  is  the  extent  to  which  an  APSE  facilitates 
evaluation  of  its  own  performance,  so  that  productivity  and  quality  metrics  can  be  gathered 
and  analyzed.  It  is  through  the  use  of  this  attribute  that  the  extensibility  (see  below)  of  an 
APSE  can  be  exploited  effectively  in  a  continual  process  of  improvement. 

3.3.3  Adaptation  Attributes 

fExpandabilitv/Auamentabilitv  6.3.1/6.4.4]  —  This  is  the  extent  to  which  an  APSE 
facilitates  the  addition  of  new  capabilities  in  response  to  needs  that  go  beyond  its  original 
requirements.  The  new  capabilities  could  include  new  functions,  expanded  data  capacity,  or 
new  types  of  data  relationships.  The  word  used  to  describe  this  feature  in  several  of  the 
papers  cited  above  is  extensibility.  Given  the  current  state  of  the  art  and  rapid  pace  of 
change  of  environment  technology,  the  presence  of  this  quality  (however  it  is  achieved) 
appears  necessary  in  order  to  provide  a  path  for  evolutionary  improvements. 

flnteroperabilitv/Commonalitv  6. 3. 2/6. 4.7]  —  This  is  the  ability  of  APSEs  to 
exchange  data  base  objects  and  their  relationships  without  conversion  of  formats,  and  the  use 
of  interface  standards  (e.g.,  CAIS  [@CAIS])  to  facilitate  such  exchanges. 

fTransportabilitv  6.3.4]  —  This  is  the  extent  to  which  an  APSE  supports  the 
movement  of  software  components  to  or  from  another  APSE  without  change  in  functionality  or 
reprogramming,  and  the  use  of  interface  standards  (e.g.,  CAIS)  to  facilitate  such  movements. 

3.4  APPROACHES  TO  WHOLE-APSE  E&V 

This  section  provides  a  brief  discussion  of  approaches  to  whole-APSE 
assessment,  including  both  evaluation  of  performance  and  quality,  and  validation  of 
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conformance  to  standards.  References  to  Guidebook  sections  and  other  documents  are 
included,  in  cases  where  there  are  known  examples  of  existing  assessors  or  assessment 
products  under  development.  Such  assessors  can  be  categorized  generally  as  either  “tools" 
or  “aids.”  A  more  refined  breakdown  is  the  following: 

•  Benchmarks  and  Test  Suites  (Tools) 

•  Questionnaires  (Aids) 

•  Monitored  Experiments  (Aids  that  may  use  Tools) 

•  Decision  Aids  (Aids  that  may  use  Tools  and  other  Aids). 

These  are  discussed,  in  turn,  in  the  following  four  subsections. 

3.4.1  Renchma*,,"‘  Tes* 

Benchmarks  are  standard  tests  used  to  measure  the  execution,  performance  or 
acceptability  of  an'APSE  function  or  set  of  functions.  A  test  suite  is  an  organized  collection  of 
such  tests.  The  Ada  Compiler  Validation  Capability  [@ACVC  1986]  is  a  test  suite  designed  to 
test  conformance  of  an  Ada  compiler  to  the  formal  definition  of  the  language  [@DoD  1983].  A 
prototype  compiler  performance  evaluation  test  suite  has  been  generated  by  the  Institute  for 
Defense  Analysis  [@GB:  5.2],  and  a  more  carefully  engineered  set  known  as  the  ACEC  or 
Ada  Compiler  Evaluation  Capability  [@GB:  5.3]  is  being  developed,  under  an  E&V  Task 
contract,  by  the  Boeing  Company.  Another  collection  of  Ada  compiler  performance  tests  has 
been  gathered  by  the  ACM  SIGAda  Performance  Issues  Working  Group  [@PIWG  1987]. 
Results  of  these  tests,  run  on  a  number  of  commercial  compilers,  will  be  published  in  the  open 
literature. 

3.4.2  Questionnaires 

Questionnaires  are  used  to  gather  data  from  vendors  or  users  of  tools  and 
APSEs.  Examples  of  such  data  might  include  specification  parameters,  design  features, 
historical  information,  typical  usage  scenarios,  implementation  strategies,  enhancement  plans 
or  desires,  and  problem  reports.  An  early  example,  in  Ada  terms,  of  such  a  questionnaire 
was  one  applied  to  the  evaluation  of  the  ROLM  Ada  Work  Center  [@Castor  1983]. 

A  whole-APSE-oriented  example  of  the  use  of  questionnaires  is  the  book 
"Selecting  an  Ada  Environment"  [@Lyons  and  Nissen  1986],  which  is  synopsized  in  the  E&V 
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Guidebook  [@GB:  4.9].  The  book  is  organized  around  background  discussions  of  various 
topics  followed  by  appropriate  questions  addressing  each  topic.  The  final  chapter  begins  with 
the  following  set  of  high-level  questions  appropriate  for  a  potential  purchaser  of  an  APSE: 

a)  What  does  the  environment  consist  of? 

b)  In  particular,  what  tools  are  supplied? 

c)  What  are  the  deliverables? 

d)  What  does  it  cost? 

e)  What  support  is  available? 

f)  Can  extra  tools  be  added  to  the  environment  easily? 

g)  What  are  the  conditions  of  use  (number  of  users,  involvement  of 
third  parties,  etc)? 

h)  What  hardware  and  software  resources  (including  licenses)  are 
'  needed  to  support  the  environment?" 

More  detailed  questions  follow,  under  specific  headings  such  as  support,  interfaces,  and  other 
issues. 


3.4.3  Monitored  Experiments 

Monitored  experiments,  based  on  model  projects  involving  an  aggregation  of 
APSE  functions  or  tools,  can  be  performed  using  APSEs  or  APSE  components  to  gather  data 
in  a  systematic  and  controlled  manner.  These  experiments  can  be  used  for  both  qualitative 
and  quantitative  assessments  of  the  functionality,  usability,  and  performance,  as  well  as  for 
the  more  informal  characteristics  of  APSEs. 

3.4.4  Decision  Aids 

Decision  aids  allow  a  user  to  assess  an  APSE  from  a  particular  point  of  view. 
Decision  aids  may  combine  the  results  of  a  number  of  tests,  questionnaires,  and/or  monitored 
experiments,  each  of  which  is  weighted  according  to  its  value  for  the  view  being  considered 
[@GB  3]. 
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4.  LIFE  CYCLE  ACTIVITIES 


This  chapter  deals  with  the  life  cycle  activities  served  by  the  support  environment. 
The  foundation  for  this  discussion  is  DoD-STD-2167A  [@DoD-STD-2167A].  The  divisions  of  the 
life  cycle  that  are  chosen  to  represent  the  progression  of  activities  depend  on  the  view  that  _a 
person  has  of  the  APSE.  If  one  thinks  of  the  APSE  as  only  providing  an  environment  for 
developing  software,  then  the  activity  groups  can  be  limited  to  the  following  six: 

•  Software  Requirements  Analysis 

•  Preliminary  Design 

•  Detailed  Design 

•  Coding  and  CSU  Testing 

•  CSC  Integration  and  Testing 

•  CSCI  Testing. 

However,  if  one  sees  the  APSE  as  providing  support  across  the  entire  system  life 
cycle  (Integrated  Project  Support  Environment,  or  IPSE,  rather  than  APSE)  [Whole  APSE 
Issues  3.],  then  the  following  five  activity  groups  should  be  added  as  well: 

•  System  Concepts 

•  System  Requirements  Analysis/Design 

•  System  Integration  and  Testing 

•  Operational  Testing  and  Evaluation 

•  Change  Requirements. 

Of  the  above  five  activity  groups,  the  first  two  precede  the  six  software 
development  phases  and  the  next  two  follow  the  six  groups  in  the  overall  system  life  cycle. 
The  last  activity  group,  change  requirements,  is  one  that  may  be  started  in  parallel  with  any  of 
the  other  groups  and  addresses  the  issues  of  enhancement,  error  correction,  and 
modification. 
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Also,  a  global  life  cycle  ‘activity  group’  is  introduced.  TbisJs.QQ.La. true  portion  Qf 
the  life  cycle,  but  represents  a  group  of  activities  that  mav  occur  anywhere  across  the  entire 
life  cycle.  An  example  of  a  global  function  is  general  purpose  text  editing. 

Chapter  4  of  DoD-STD-2167A  lists  the  general  requirements  for  the  life  cycle 
activities.  These  requirements  fall  into  the  following  activity  classes: 

•  Software  Development  Management 

•  Software  Engineering 

•  Formal  Qualification  Testing 

•  Software  Product  Evaluations 

•  Software  Configuration  Management 

•  Transitioning  to  Software  Support. 

These  activities  are  used  as  a  second-level  of  classification  under  each  top-level  life  cycle, 
activity  division  of  this  chapter. 

Figure  4-1  shows  the  relationships  between  the  life  cycle  activities  and  the  other 
elements  in  the  Reference  Manual.  Each  of  the  life  cycle  activity  groups  are  described  in  the 
following  sections  and  under  each  section  are  six  subsections  that  deal  with  each  of  the 
activity  classes.  The  functions  specified  for  each  section  are  those  that  are  typically  used  in 
that  portion  of  the  life  cycle. 
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Figure  4-1  Life  Cycle  Activity  Relationships 
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4.1  SYSTEM  CONCEPTS 

4.1.1  System  Development  Management 

Products: 

Functions: 

4.1.2  System  Engineering 

Products: 


Functions: 

[Requirements  Simulation  7.3.2. 1, 

Requirements  Prototyping  7.3.2.2. 

Simulation  and  Modeling  7.3.2.3, 

Emulation  7.3.2.13] 


4.1.3  Formal  Qualification  Testing 

Products: 

Functions: 
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4.1.4  System  Product  Evaluations 

Products: 

Functions: 


4.1.5  Configuration  Management 

Products: 

Functions: 


4.1.6  Transitioning  to  System  Support 

Products: 

Functions: 
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4.2  SYSTEM  REQUIREMENTS  ANALYSIS/DESIGN 


4.2.1  System  Development  Management 


Products: 

[Software  Development  Plan  (SDP)] 

Functions: 

[Predefined  and  User-Defined  Forms  7. 1.2.3] 


4.2.2  System  Engineering 


Products: 

[System/Segment  Specification  (SSS), 

Prime  Item  Development  Specification  (PIDS), 

Critical  Item  Development  Specification  (CIDS), 

System/Segment  Design  Document  (SSDD), 

Software  Requirements  Specification  (SRS), 

Interface  Requirements  Specification  (IRS)] 

Functions: 

[Predefined  and  User-Defined  Forms  7. 1.2.3, 

Systems  Requirements  7. 1.6.1, 

Requirements  to  Natural  Language  7.1. 6.3, 

Requirements  Prototyping  7.3.2.2] 


4.2.3  Formal  Qualification  Testing 

Products: 

Functions: 
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4.2.4  System  Product  Evaluations 

Products: 

Functions: 

[Requirements  Simulation 

4.2.5  Configuration  Management 

Products: 

Functions: 


4.2.6  Transitioning  to  System  Support 


Products: 

[Computer  Resources  Integrated  Support  Document 
Functions: 

[Predefined  and  User-Defined  Forms 


7.3.2. 1] 


(CRISD)] 

7.1.2.31 
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4.3  SOFTWARE  REQUIREMENTS  ANALYSIS 


4.3.1  System  Development  Management 


Products: 

[Software  Development  Plan 


Functions: 

[Predefined  and  User-Defined  Forms 
Resource  Estimation 


4.3.2  Software  Engineering 


Products: 

[Software  Requirements  Specification 
Interface  Requirements  Specification 


Functions: 

[Predefined  and  User-Defined  Forms 
Software  Requirements 
Requirements  to  Natural  Language 
Program  Library  Management 
Requirements  Prototyping 


4.3.3  Formal  Qualification  Testing 

Products: 

Functions: 


(SDP)] 


7.1. 2.3, 
7.2. 2. 5] 


(SRS), 

(IRS)] 


7.1. 2.3, 

7.1. 6.2, 

7.1. 6.3, 
7.2.1. 7, 
7.3. 2.2] 
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4.3.4  Software  Product  Evaluations 


Products: 

Functions: 

[Tracking 

Data  Flow  Analysis 
Functional  Analysis 
Interface  Analysis 
Traceability  Analysis 
Testability  Analysis 
Test  Condition  Analysis 
Quality  Analysis 
Requirements  Simulation 


4.3.5  Configuration  Management 

Products: 

Functions: 

[Specification  Management 


4.3.6  Transitioning  to  Software  Support 


Products: 

[Computer  Resources  Integrated  Support  Document 
Functions: 

[Predefined  and  User-Defined  Forms 


7. 2. 2. 6, 

7.3.1. 3, 

7.3.1. 4, 

7.3.1. 5, 

7.3.1. 6, 

7.3.1. 7, 

7.3.1. 8, 

7.3.1. 9, 
7.3. 2.1] 


7.2. 1.6] 


(CRISD)] 


7. 1.2.3] 
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4.4  PRELIMINARY  DESIGN 


4.4.1  Software  Development  Management 


Products: 


[Software  Development  Plan 

(SDP)] 

Functions: 

[Predefined  and  User-Defined  Forms 

Resource  Estimation 

7.1. 2.3, 
7.2.2.5] 

4.4.2  Software  Engineering 

Products: 

[Software  Design  Document 

Interface  Design  Document 

(SDD), 

(IDO)] 

Functions: 

[Predefined  and  User-Defined  Forms 
Preliminary  Design 

Design  Generation 

Program  Library  Management 

Design  Prototyping 

7. 1.2.3, 

7.1. 6.4, 
7.1. 7.1, 
7.2. 1.7, 
7.3.2.4] 

4.4.3  Formal  Qualification  Testing 

Products: 

[Software  Test  Plan 

(STP)l 

Functions: 

[Predefined  and  User-Defined  Forms 

Formal  Verification 

7.1. 2.3, 
7.3.31 
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4.4.4  Software  Product  Evaluations 


Products: 

Functions: 

[Requirements  Reconstruction 
Tracking 

Interface  Analysis 
Quality  Analysis 
Complexity  Measurement 
Completeness  Checking 
Consistency  Checking 
Cross  Reference 
Invocation  Analysis 
Scanning 

Structured  Walkthrough 
Simulation  and  Modeling 


4.4.5  Configuration  Management 

Products: 

Functions: 

[Specification  Management 


4.4.6  Transitioning  to  Software  Support 


Products: 

[Computer  Resources  Integrated  Support  Document 
Computer  System  Operator’s  Manual 
Software  User’s  Manual 

Functions: 

[Predefined  and  User-Defined  Forms 


7.1. 7.2, 
7. 2. 2. 6, 
7.3. 1.5, 

7. 3. 1.9, 

7.3.1.10, 

7.3.1.12, 

7.3.1.13, 
7.3.1.17, 

7.3.1.19, 

7.3.1.20, 

7.3.1.21, 
7.3. 2.3] 


7.2.1. 6] 


(CRISD), 
(CSOM) , 
(SUM)] 


7.1. 2.3] 


4-10 


E&V  Reference  Manual,  Version  1.1 


4.5  DETAILED  DESIGN 


4.5.1  Software  Development  Management 


Products: 

[Software  Development  Plan 
Functions: 

[Predefined  and  User-Defined  Forms 
Resource  Estimation 


4.5.2  Software  Engineering 


Products: 

[Software  Design  Document 
Interface  Design  Document 
Software  Development  File 

Functions: 

[Predefined  and  User-Defined  Forms 
Detailed  Design 
Design  Generation 
Program  Library  Management 
Design  Prototyping 


4.5.3  Formal  Qualification  Testing 


Products: 

[Software  Test  Description 
Functions: 

[Predefined  and  User-Defined  Forms 
Formal  Verification 


(SDP)] 


7.1. 2.3, 
7.2.2. 5] 


(SDD), 

(IDD), 

(SDF)] 


7. 1.2.3, 
7.1. 6.5, 
7.1. 7.1, 
7.2.1. 7, 
7. 3. 2. 4] 


(STD)] 


7.1. 2.3, 
7.3.3] 
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4.5.4  Software  Product  Evaluations 


Products: 


Functions: 

[Requirements  Reconstruction 
Tracking 

Interface  Analysis 
Quality  Analysis 
Complexity  Measurement 
Completeness  Checking 
Consistency  Checking 
Cross  Reference 
Invocation  Analysis 
Scanning 

Structured  Walkthrough 
Simulation  and  Modeling 
Symbolic  Execution 


7.1. 7.2, 
7.2.2.6, 
7.3.1. 5, 

7.3.1. 9, 

7.3.1.10, 
7.3.1.12. 
7.3.1.13, 
7.3.1.17, 

7.3.1.19, 

7.3.1.20, 

7.3.1.21, 

7.3.2.3, 
7.3.4] 


4.5.5  Configuration  Management 


Products: 


Functions: 

[Specification  Management 


7.2. 1.6] 


4.5.6  Transitioning  to  Software  Support 


Products: 

[Computer  Resources  integrated  Support  Document 
Computer  System  Operator’s  Manual 
Software  User’s  Manual 
Software  Programmer’s  Manual 
Firmware  Support  Manual 


(CRISD) , 

(CSOM), 

(SUM), 

(SPM), 

(FSM)] 


Functions: 

[Predefined  and  User-Defined  Forms 


7. 1.2.3] 
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4.6  CODING  AND  CSU  TESTING 


4.6.1  Software  Development  Management 

Products: 

[Software  Development  Plan 
Functions: 

[Predefined  and  User-Defined  Forms 
Resource  Estimation 


4.6.2  Software  Engineering 

Products: 

[Software  Development  File 
Functions: 

[Predefined  and  User-Defined  Forms 
Assembling 
Compilation 
Conversion 
Macro  Expansion 
Structure  Preprocessing 
Body  Stub  Generation 
Preamble  Generation 
Linking/Loading 
Interpretation 
Program  Generation 
Program  Library  Management 
Runtime  Environment 


4.6.3  Formal  Qualification  Testing 


Products: 

[Software  Test  Description 


Functions: 

[Predefined  and  User-Defined  Forms 
Test  Data  Management 
Formal  Verification 


(SDP)] 


7.1. 2.3, 
7.2.2. 5] 


(SDF)] 


7. 1.2.3, 

7.1. 6.6, 

7.1. 6.7, 

7. 1.6.8, 

7.1. 6.9, 

7.1.6.10, 

7.1.6.11, 

7.1.6.12, 

7.1.6.13, 

7.1.6.14, 

7.1. 7.3, 
7.2. 1.7, 
7.2.3.5] 


(STD)] 


7.1. 2.3, 
7.2. 1.8, 
7.3.3] 
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4.6.4  Software  Product  Evaluations 


Products: 


Functions: 

[Requirements  Reconstruction 
Source  Reconstruction 
Decompilation 
Disassembling 
Tracking 

Data  Flow  Analysis 
Interface  Analysis 
Quality  Analysis 
Complexity  Measurement 
Completeness  Checking 
Consistency  Checking 
Cross  Reference 
Maintainability  Analysis 
Invocation  Analysis 
Scanning 

Structured  Walkthrough 

Auditing 

Error  Checking 

Statistical  Analysis 

Statistical  Profiling 

Structure  Checking 

Type  Analysis 

Units  Analysis 

I/O  Specification  Analysis 

Sizing  Analysis 

Debugging 

Executable  Assertion  Checking 
Constraint  Evaluation  (Contention) 
Coverage/Frequency  Analysis 
Mutation  Analysis 
Testing 

Regression  Testing 
Resource  Utilization 
Emulation 
Timing  Analysis 
Tuning 

Reliability  Analysis 
Real  Time  Analysis 
Symbolic  Execution 


7.1. 7.2, 

7.1. 7.4, 

7.1. 7.5, 

7.1. 7.6, 

7. 2. 2.6, 

7.3.1. 3, 

7.3.1. 5, 

7.3.1. 9. 

7.3.1.10, 

7.3.1.12, 

7.3.1.13, 

7.3.1.17, 

7.3.1.18, 

7.3.1.19, 

7.3.1.20, 

7.3.1.21, 

7.3.1.22, 

7.3.1.23, 

7.3.1.24, 

7.3.1.25, 

7.3.1.26, 

7.3.1.27, 

7.3.1.28, 

7.3.1.29, 

7.3.1.30, 

7.3.2.5, 

7.3. 2.6, 

7.3. 2. 7, 

7.3. 2. 8, 

7.3.2.9, 

7.3.2.10, 

7.3.2.11, 

7.3.2.12, 

7.3.2.13, 

7.3.2.14, 

7.3.2.15, 

7.3.2.16, 

7.3.2.17, 
7.3.4] 
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4.6.5  Configuration  Management 

Products: 

Functions: 


4.6.6  Transitioning  to  Software  Support 


Products: 

[Computer  System  Operator’s  Manual 
Software  User’s  Manual 
Software  Programmer’s  Manual 
Firmware  Support  Manual 

Functions: 

[Predefined  and  User-Defined  Forms 
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(SUM), 

(SPM), 

(FSM)] 
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4.7  CSC  INTEGRATION  AND  TESTING 

4.7.1  Software  Development  Management 

Products: 

[Software  Development  Plan  (SDP)] 

Functions: 

[Predefined  and  User-Defined  Forms  7. 1.2.3, 

Resource  Estimation  7.2.2.S] 


4.7.2  Software  Engineering 

Products: 

[Software  Development  File  (SDF)  ] 

Functions: 

[Predefined  and  User-Defined  Forms  7. 1.2.3, 

Assembling  7. 1.6.6, 

Compilation  7. 1.6.7, 

Conversion  7. 1.6.8, 

Macro  Expansion  7. 1.6.9, 

Structure  Preprocessing  7.1.6.10, 

Body  Stub  Generation  7.1.6.11, 

Preamble  Generation  7.1.6.12, 

Unking/Loading  7.1.6.13, 

Interpretation  7.1.6.14, 

Program  Library  Management  7.2. 1.7, 

Runtime  Environment  7.2.3.S] 


4.7.3  Formal  Qualification  Testing 

Products: 

[Software  Test  Description  (STD)] 

Functions: 

[Predefined  and  User-Defined  Forms  7.1. 2.3, 

Test  Data  Management  7.2. 1.8] 
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4.7.4  Software  Product  Evaluations 


Products: 


Functions: 


[Requirements  Reconstruction 

7.1. 7.2. 

Source  Reconstruction 

7. 1.7.4, 

Decompilation 

7.1. 7.5, 

Disassembling 

7.1. 7.6. 

Tracking 

7.2. 2. 6, 

Data  Flow  Analysis 

7.3.1. 3. 

Interface  Analysis 

7.3.1. 5, 

Quality  Analysis 

7.3. 1.9, 

Complexity  Measurement 

7.3.1.10, 

Completeness  Checking 

7.3.1.12, 

Consistency  Checking 

7.3.1.13, 

Cross  Reference 

7.3.1.17, 

Maintainability  Analysis 

7.3.1.18, 

Scanning 

7.3.1.20, 

Structured  Walkthrough 

7.3.1.21, 

Auditing 

7.3.1.22, 

Error  Checking 

7.3.1.23, 

Statistical  Profiling 

7.3.1.25, 

Structure  Checking 

7.3.1.26, 

Type  Analysis 

7.3.1.27, 

Units  Analysis 

7.3.1.28, 

I/O  Specification  Analysis 

7.3.1.29, 

Sizing  Analysis 

7.3.1.30, 

Debugging 

7.3.2.S, 

Executable  Assertion  Checking 

7.3.2.6, 

Constraint  Evaluation  (Contention) 

7.3.2 .7, 

Coverage/Frequency  Analysis 

7.3.2.8, 

Mutation  Analysis 

7. 3. 2. 9, 

Testing 

7.3.2.10, 

Regression  Testing 

7.3.2.11, 

Resource  Utilization 

7.3.2.12, 

Emulation 

7.3.2.13, 

Timing  Analysis 

7.3.2.14, 

Tuning 

7.3.2.15, 

Reliability  Analysis 

7.3.2.16, 

Real  Time  Analysis 

7.3.2.17, 

Symbolic  Execution 

7.3.4, 

Problem  Report  Analysis 

7.3.5] 
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4.7.5  Configuration  Management 

Products: 

Functions: 


4.7.6  Transitioning  to  Software  Support 


Products: 

[Computer  System  Operator’s  Manual 
Software  User’s  Manual 
Software  Programmer’s  Manual 
Firmware  Support  Manual 

Functions: 

[Predefined  and  User-Defined  Forms 


(CSOM), 

(SUM), 

(SPM), 

(FSM)] 


7. 1.2.3] 
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4.8  CSCI  TESTING 


4.8.1  Software  Development  Management 

Products: 

[Software  Development  Plan  (SDP)] 

Functions: 

[Predefined  and  User-Defined  Forms  7. 1.2.3, 

Resource  Estimation  7.2.2.S] 


4.8.2  Software  Engineering 

Products: 

[Software  Design  Document  (SDD), 

Interface  Design  Document  (IDD), 

Software  Development  File  (SDF), 

Software  Product  Specification  (SPS)] 

Functions: 

[Predefined  and  User-Defined  Forms  7. 1.2.3, 

Assembling  7. 1.6.6, 

Compilation  7. 1.6.7, 

Conversion  7. 1.6.8, 

Macro  Expansion  7. 1.6.9, 

Structure  Preprocessing  7.1.6.10, 

Preamble  Generation  7.1.6.12, 

Linking/Loading  7.1.6.13, 

Interpretation  7.1.6.14, 

Program  Library  Management  7.2. 1.7, 

Runtime  Environment  7.2. 3.5, 

Import/Export  7.2.3. 6] 
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4.8.3  Formal  Qualification  Testing 


Products: 

(Software  Test  Description 
Software  Test  Report 


Functions: 

[Predefined  and  User-Defined  Forms 
Test  Data  Management 
Quality  Analysis 
Coverage/Frequency  Analysis 
Testing 

Regression  Testing 
Resource  Utilization 
Emulation 
Timing  Analysis 
Tuning 

Reliability  Analysis 
Real  Time  Analysis 
Problem  Report  Analysis 


4.8.4  Software  Product  Evaluations 


Products: 

Functions: 

(Tracking 

Debugging 


4.8.5  Configuration  Management 


Products: 

[Version  Description  Document 
Functions: 

[Predefined  and  User-Defined  Forms 


(STD), 

(STR)] 


7.1. 2.3, 

7. 2. 1.8, 

7.3.1. 9, 
7. 3. 2. 8, 

7.3.2.10, 

7.3.2.11, 

7.3.2.12, 

7.3.2.13, 

7.3.2.14, 

7.3.2.15, 

7.3.2.16, 

7.3.2.17, 
7.3.5] 


7.2. 2.6, 
7. 3. 2. 5] 


(VDD)] 


7. 1.2.3] 
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4.8.6  Transitioning  to  Software  Support 


Products: 

[Computer  System  Operator’s  Manual  (CSOM), 

Software  User’s  Manual  (SUM), 

Software  Programmer’s  Manual  (SPM), 

Firmware  Support  Manual  (FSM)j 

Functions: 

[Predefined  and  User-Defined  Forms  7. 1.2.3] 
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4.9  SYSTEM  INTEGRATION  AND  TESTING 


4.9.1  Software  Development  Management 

Products: 

Functions: 


4.9.2  System  Engineering 

Products: 

[Software  Product  Specification 
Functions: 

[Predefined  and  User-Defined  Forms 
Assembling 
Compilation 
Conversion 
Macro  Expansion 
Structure  Preprocessing 
Preamble  Generation 
Linking/Loading 
Interpretation 

Program  Library  Management 
Runtime  Environment 
Import/Export 


4.9.3  Formal  Qualification  Testing 


Products: 

Functions: 

[Import/Export 
Problem  Report  Analysis 


(SPS)] 


7. 1.2.3, 

7. 1.6.6, 

7.1. 6.7, 

7.1. 6.8, 

7.1. 6.9, 

7.1.6.10, 

7.1.6.12, 

7.1.6.13, 

7.1.6.14, 
7.2.1. 7, 
7.2. 3. 5, 
7.2.3.63 


7. 2. 3. 6, 
7.3.5] 
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4.9.4  System  Product  Evaluations 

Products: 

Functions: 

4.9.5  Configuration  Management 

Products: 

[Version  Description  Document 
Functions: 

[Predefined  and  User-Defined  Forms 

4.9.6  Transitioning  to  System  Support 

Products: 

Functions: 


(VDD)] 
7.1. 2.3] 
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4.10  OPERATIONAL  TESTING  AND  EVALUATION 

4.10.1  Software  Development  Management 

Products: 

Functions: 


4.10.2  System  Engineering 


Products: 


Functions: 


[Assembling 

7.1. 6.6, 

Compilation 

7.1. 6.7. 

Conversion 

7.1. 6. 8, 

Macro  Expansion 

7.1. 6.9, 

Structure  Preprocessing 

7.1.6.10, 

Preamble  Generation 

7.1.6.12, 

Linking/Loading 

7.1.6.13, 

Interpretation 

7.1.6.14, 

Program  Library  Management 

7.2.1. 7, 

Runtime  Environment 

7.2. 3.5, 

Import/Export 

7. 2. 3. 6] 

4.10.3  Formal  Qualification  Testing 


Products: 

Functions: 

[Import/Export 
Problem  Report  Analysis 


7. 2. 3. 6. 
7.3.5} 
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4.10.4  System  Product  Evaluations 

Products: 

Functions: 


4.10.5  Configuration  Management 

Products: 

Functions: 

[Evaluation  Results  Management 

4.10.6  Transitioning  to  System  Support 

Products: 

Functions: 


7.2. 1.9] 
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4.11  CHANGE  REQUIREMENTS 


4.11.1  System  Development  Management 

Products: 

Functions: 


4.11.2  System  Engineering 


Products: 

[Engineering  Change  Proposal 
Functions: 

[Predefined  and  User-Defined  Forms 
Change  Impact  Analysis 


4.11.3  Formal  Qualification  Testing 

Products: 

Functions: 


(ECP)] 

7.1. 2.3, 
7.3.6. 1] 
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4.11.4  System  Product  Evaluations 

Products: 

Functions: 


4.11.5  Configuration  Management 


Products: 

[Specification  Change  Notice  (SCN)] 


Functions: 

[Predefined  and  User-Defined  Forms  7. 1.2.3] 


4.11.6  Transitioning  to  System  Support 


Products: 

Functions: 


4-27 


E&V  Reference  Manual,  Version  1.1 


4.12  GLOBAL 


4.12.1  System  Development  Management 


Products: 

Functions: 

[Data  Base  (Object)  Management 
Documentation  Management 
File  Management 
Electronic  Mail 
Electronic  Conferencing 
Cost  Estimation 
Quality  Specification 
Scheduling 

Work  Breakdown  Structure 


4.12.2  System  Engineering 


Products: 

Functions: 

[Text 
Graphics 
MIL-STD  Format 
Table  of  Contents 
On-Line  Assistance  Processing 
Sort/Merge 
Graphics  Generation 
Command  Language  Processing 
Input/Output  Support 
Kernel 

Import/Export 


7.2.1. 1, 

7.2.1. 2, 

7.2.1. 3, 

7.2.1. 4, 

7.2.1. 5, 

7.2.2. 1, 

1. 2.2.2, 
7.2.2.3, 
7.2.2.4] 


7.1. 1.1. 

7.1. 1.3, 

7.1 .2.1 . 

7.1. 2.2. 

7.1.3, 

7.1.4, 

7.1.5, 

7. 2. 3.1, 

7. 2. 3. 2, 

7.2.3.3, 
7.2.3.61 
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4.12.3  Formal  Qualification  Testing 


Products: 

Functions: 


4.12.4  System  Product  Evaluations 


Products: 

Functions: 
[Comparison 
Spelling  Checking 


4.12.5  Configuration  Management 

Products: 

Functions: 

[Configuration  Management 


4.12.6  Transitioning  to  System  Support 

Products: 


7.3.1. 1, 
7.3.1. 2] 


7. 2.2.7] 


Functions: 


E&V  Reference  Manual,  Version  1.1 


5. 


APSE  TOOL  CATEGORIES 


This  chapter  deals  with  APSEs,  toolsets,  and  tools.  E&V  technology  is  always 
applied  to  APSEs  or  their  components  and  thus  this  index  will  be  a  natural  starting  point  for 
many  users  of  the  RM.  The  following  sections  describe  the  APSE  and  its  components  and 
relate  the  components  to  functions  as  shown  in  Fig.  5-1 .  The  relationships  between  tools  and 
functions  given  in  this  chapter  are  typical  (or  traditional)  relationships.  The  real  capabilities 
should  be  determined  bv  the  tool  specifications,  marketing  claims,  or  the  like. 

There  is  also  a  relationship  between  tools  and  attributes.  Some  attributes  are 
related  to  qualities  of  the  function (s)  performed  (such  as  completeness)  by  the  software,  while 
other  attributes  relate  to  non-functional  aspects  (such  as  modularity)  of  the  software.  Thus,  to 
determine  how  to  assess  a  tool,  both  the  tool-function  relationships  and  the  attributes_musLbe 
explored  .tQ-idgntify..the..metijQ.sJQJae,  .used- 

APSEs  are  the  highest  level  of  toolset.  APSEs  should  contain  all  the  tools  and 
toolsets  needed  to  support  the  full  spectrum  of  project  activities.  APSEs.  like  toolsets,  have 
characteristics  that  mav  be  assessed  as  a  whole  fWhole  APSE  Issues  3J.  Toolsets  are  one 
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Figure  5-1  Tool  Relationships 
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or  more  tools  which  are  intimately  coupled  and  support  a  related  set  of  functions  within  the 
APSE.  Tools  are  the  smallest  APSE  components  which  can  be  independently  acquired.  Each 
section  in  this  chapter  describes  a  toolset  category  that  might  be  found  in  an  APSE.  The 
section  titles  are: 


•  Computer  Management  System 

•  Project  Management  System 

•  Compilation  System 

•  Document  System 

•  Desktop  System 

•  Static  Analyzer  System 

•  Dynamic  Analyzer  System. 

The  subsections  within  each  section  give  the  tool  categories  that  comprise  the  toolset.  The 
complete,  two-level  tool  taxonomy  may  be  seen  by  examining  the  Table  of  Contents  under 
Chapter  5.  Not  all  toolsets  are  necessary  for  an  APSE:  likewise,  for  a  specific  toolset,  all  tools 
may  not  always  be  present. 
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5.1  COMPUTER  MANAGEMENT  SYSTEM 


Description: 

Those  tools  needed  to  access,  use,  and  maintain  the  hardware  and  software  which 
consists  of  the  APSE  and  the  system  on  which  it  runs. 


5.1.1  Command  Language  Processor 
Cross  References; 

Functions: 

[Command  Language  Processing  7.2.3. 1] 

5.1.2  Archive,  Backup,  and  Retrieval  System 

Cross  References; 

Functions: 

[Import/Export  7.2. 3.6] 


5.1.3  Security  System 
Cross  References: 

Functions: 

[Kernel  7.2. 3.3] 
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5.1.4  Job  Scheduler 
Cross  References; 

Functions: 

[Runtime  Environment  7.2.3.5J 


5.1.5  Resource  Controller 
Cross  References: 


Functions: 

[Runtime  Environment 


7.2.3.5] 


5.1.6  File  Manager 

Cross  References;. 


Functions: 

[File  Management 


7.2. 1.3] 


5.1.7  Import/Export  System 
Cross  References: 


Functions: 

[Import/Export 


7.2.3.61 
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5.1.8  On-Line  Assistance  Processor 

Crq>.ss..fie.fgreDce.§j 

Functions: 

[On-Line  Assistance  Processing 

5.1.9  Data  Base  Manager 
Cross  References: 

Functions: 

[Data  Base  Management 


7.1.3] 


7.2.1. 1] 
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5.2  PROJECT  MANAGEMENT  SYSTEM 


Description: 

Those  tools  needed  to  plan,  develop,  and  maintain  an  applications  system. 


5.2.1  Cost  Estimator 
Cross  References: 

Functions: 

[Cost  Estimation  7.2.2. 1] 


5.2.2  Quality  Analyzer 

Cross  References; 


Functions: 

[Quality  Specification  7.2. 2.2, 

Quality  Assessment  7.2.2.S] 


5.2.3  Scheduler 
Cross  References; 


Functions: 

[Scheduling 


7.2.2.3J 
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5.2.4  Work  Breakdown  Structure 
Cross  References: 

Fi 'notions: 

[Work  Breakdown  Structure 

5.2.5  Resource  Estimator 
Cross  References: 

Functions: 

[Resource  Estimation 

5.2.6  Tracking 

Cross-Rfifatonseai 

Functions: 

[Tracking 

5.2.7  Configuration  Manager 
Cross  References; 

Functions: 

[Configuration  Management 


Version  1.1 


7.2.2. 4] 


7. 2. 2. 5] 


7.2.2.6] 


7.2.2.7] 
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5.2.8  Problem  Report  Analyzer 


Cross  References: 


Functions: 

[Problem  Report  Analysis 


5.2.9  Change  Request  Analyzer 

Cross  References; 


Functions: 

[Change  Request  Analysis 


7.3.5] 


7.3.6] 
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5.3  COMPILATION  SYSTEM 


CteSfiriplIgni 

Those  tools  needed  to  produce  and  run  software  systems. 


5.3.1  Program  Library  Manager 
Cross  References; 


Functions: 

[Program  Library  Management  7.2. 1.7] 

5.3.2  Syntax-Directed  Editor 

Cross-References; 


Functions: 

[Text  Editing  7. 1.1.1, 

Syntax  And  Semantics  Checking  7.3.1.15] 


5.3.3  Compiler 

Cross  .References; 


Functions: 

[Compilation 


7.1. 6.7] 
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5.3.4  Assembler 
Cross  References: 

Functions: 

[Assembling  7. 1.6.6] 


5.3.5  Linker 

Cross  References: 


Functions: 

[Linking/Loading 


7.1.6.13] 


5.3.6  Loader 

Cross  References; 


Functions: 

[Linking/Loading 


7.1.6.13] 


5.3.7  Interpreter 
Cross  References: 


Functions: 

[Interpretation 


7.1.6.14] 
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5.3.8  Runtime  Library 
Cross  References: 


Functions: 

[Input/Output  Support  7.2.3.2, 

Math/Statistics  7.2.3.4, 

Runtime  Environment  7.2.3. 5] 
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5.4  DOCUMENT  SYSTEM 


Description: 

Those  tools  needed  to  develop  and  produce  documents. 


5.4.1  Document  Manager 
Cross  References: 

Functions: 

[Document  Management  7.2. 1 .2] 


5.4.2  Word  Processor 

Grass  References; 


Functions: 

[Text  Editing  7. 1.1.1, 

Syntax  And  Semantics  Checking  7.3.1.15] 


5.4.3  Spell  Checker 

Cross  References; 


Functions: 

[Spelling  Checking 


7.3. 1.2] 
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5.4.4  Graphics  Generator 
Cross  References; 


Functions: 

[Graphics  Generation 


5.4.5  Formatter 
Cross  References; 


Functions: 
[Formatting  - 
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5.5  DESKTOP  SYSTEM 


Description: 

Those  tools  needed  to  do  the  every-day,  administrative  tasks  of  business. 


5.5.1  Spreadsheet 
Cross  References: 


Functions: 

[Formatting.  7.1.2, 

Math/Statistics  7.2.3.4] 


5.5.2  Calculator 


gr.os.SLRfifftreiafi.es; 


Functions: 

[Math/Statistics  7.2.3.4J 


5.5.3  Address  Book 

Cross-References; 


Functions: 

[Sort/Merge 


7.1.4] 
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5.5.4  Electronic  Mail 
Cross  References: 

Functions: 

[Electronic  Mail  7.2. 1.4] 

5.5.5  Phone  Book 
Cross  References: 

Functions: 

[Sort/Merge  7.1.4] 


5.5.6  Electronic  Conferencing 


Ccasa-ReffiLeDOfiSL 


Functions: 

[Electronic  Conferencing 


7.2.1. 5] 


5.5.7  Calendar 


Cross  Ref srsncss; 


Functions: 

[Scheduling 


7.2. 2. 3] 
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5.5.8  Dictionary 
Cross  References! 

Functions: 

[Sort/Merge  7.\A\ 
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5.6  STATIC  ANALYZER  SYSTEM 


Description; 

Those  tools  needed  to  do  analysis  of  software  systems  without  actually  executing  the 
program(s)  being  analyzed. 


5.6.1  Comparator 
Cross  References: 

Functions: 

[Comparison  7.3. 1.1] 

5.6.2  Data  Flow  Analyzer 
Cross  References: 

Functions: 

[Data  Flow  Analysis  7.3. 1.3] 

5.6.3  Functional  Analyzer 
Cross  References: 

Functions: 

[Functional  Analysis  7.3. 1.4] 
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5.6.4  Interface  Analyzer 

Cross  References: 

Functions: 

[Interface  Analysis 


5.6.5  Traceability  Analyzer 
Cross  References: 

Functions: 

[Traceability  Analysis 


5.6.6  Testability  Analyzer 


Cross  References: 


Functions: 

[Testability  Analysis 


5.6.7  Test  Condition  Analyzer 
Cross  References: 


Functions: 

[Test  Condition  Analysis 


7.3.1. 5] 


7.3.1. 6] 


7.3.1. 7] 


7.3.1. 8] 
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5.6.8  Quality  Analyzer 

Cross  References: 

Functions: 

[Quality  Analysis 

5.6.9  Complexity  Measurer 
Cross  References; 

Functions: 

[Complexity  Measurement 

5.6.10  Correctness  Checker 
Cross  References: 

Functions: 

[Correctness  Checking 

5.6.11  Completeness  Checker 
Cross  References: 

Functions: 

[Completeness  Checking 


7.3.1. 9] 


7.3.1.10] 


7.3.1.11] 


7.3.1.12] 
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5.6.12  Consistency  Checker 
Cross  References: 

Functions: 

[Consistency  Checking  7.3.1.13] 

5.6.13  Reusability  Analyzer 
Cross  References: 

Functions: 

[Reusability  Analysis  7.3.1.14] 

5.6.14  Syntax  And  Semantics  Checker 

Cross  Fteteranassi 


Functions: 

[Syntax  And  Semantics  Checking 


7.3.1.15] 


5.6.15  Reachability  Analyzer 


Cross  References; 


Functions: 

[Reachability  Analysis 


7.3.1.16] 
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5.6.16  Cross  Referencer 
Cross  References: 


Functions: 

[Cross  Reference 


5.6.17  Maintainability  Analyzer 


Cross  References: 


Functions: 

[Maintainability  Analysis 


5.6.18  Invocation  Analyzer 

Cross  References; 


Functions: 

[Invocation  Analysis 


5.6.19  Scanner 
Cross  References; 


Functions: 

[Scanning 


7.3.1.17] 


7.3.1.18] 


7.3.1.19] 


7.3.1.20] 
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5.6.20  Structured  Walkthrough  Tool 
Cross  References: 

Functions: 

[Structured  Walkthrough 


5.6.21  Auditor 
Cross  References; 


Functions: 

[Auditing 


5.6.22  Error  Checker 

Cross  References; 


Functions: 

[Error  Checking 


5.6.23  Statistical  Analyzer 
Cross  References; 


Functions: 

[Statistical  Analysis 


# 


7.3.1.21] 


7.3.1.22] 


7.3.1.23] 


7.3.1.24] 
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5.6.24  Statistical  Profiler 
Cross  References; 


Functions: 

[Statistical  Profiling 


7.3.1.25] 


5.6.25  Structure  Checker 
Cross  References; 

Functions: 

[Structure  Checking  7.3.1.26] 


5.6.26  Type  Analyzer 


Cross  Rsforsnsss; 


Functions: 

[Type  Analysis 


7.3.1.27] 


5.6.27  Units  Analyzer 
Cross  References: 

Functions: 

[Units  Analysis  7.3.1.28] 
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5.6.28  I/O  Specification  Analyzer 
Cross  References; 


Functions: 

[I/O  Specification  Analysis 


5.6.29  Sizing  Analyzer 

Cross  References; 


Functions: 

[Sizing  Analysis 


7.3.1.29] 


7.3.1.30] 
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5.7  DYNAMIC  ANALYZER  SYSTEM 


Description: 

Those  tools  needed  to  do  analysis  of  software  systems  while  actually  executing  the 
program (s)  or  some  representation  of  the  system  being  analyzed. 


5.7.1  Requirements  Simulator 
Cross  References; 


Functions: 

[Requirements  Simulation 


7.3.2. 1] 


5.7.2  Requirements  Prototype 

Cross  References; 


Functions: 

[Requirements  Prototyping 


7.3. 2. 2] 


5.7.3  Simulation  And  Modelling  Tools 
Cross  References: 


Functions: 

[Simulation  And  Modelling 


7.3.2.3J 
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5.7.4  Design  Prototype 
Cross  References: 


Functions: 

[Design  Prototyping 


5.7.5  Debugger 
Cross  References: 


Functions: 

[Debugging 


5.7.6  Executable  Assertion  Checker 

Cross  References; 


Functions: 

[Executable  Assertion  Checking 


5.7.7  Constraint  Evaluator 
Cross  References; 


Functions: 

[Constraint  Evaluation 


7. 3. 2. 4] 


7.3. 2. 5] 


7.3. 2. 6] 


7.3.2. 7] 
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5.7.8  Coverage/Frequency  Analyzer 
Cross  References: 


Functions: 

[Coverage/Frequency  Analysis 


5.7.9  Mutation  Analyzer 
Cross  References: 


Functions: 

[Mutation  Analysis 


5.7.10  Testing  Analyzer 

Cross  References; 


Functions: 

[Testing  § 


5.7.11  Regression  Testing  Analyzer 


Cross  References: 


Functions: 

[Regression  Testing 


7.3.2.8] 


7.3.2. 9] 


7.3.2.10] 


7.3.2.11] 
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5.7.12  Resource  Utilization  Analyzer 
Cross  References; 


Functions: 

[Resource  Utilization 


5.7.13  Emulator 
Cross  References; 


Functions: 

[Emulation 


5.7.14  Timing  Analyzer 

Cross  References; 

Functions: 

[Timing  Analysis 

5.7.15  Tuning  Analyzer 

Cross  References; 

Functions: 

[Tuning 


7.3.2.12] 


7.3.2.13] 


7.3.2.14] 


7.3.2.15] 
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5.7.16  Reliability  Analyzer 
Cr.QSS.R9fg.rgnce§: 

Functions: 

[Reliability  Analysis  7.3.2.16] 

5.7.17  Real  Time  Analyzer 
Cross  References: 

Functions: 

[Real  Time  Analysis  7.3.2.17] 

5.7.18  Formal  Verification  System 

Cross  Rsfsrencss; 

Functions: 

[Formal  Verification  7.3.3] 


5.7.19  Symbolic  Execution  System 
Cross  References: 

Functions: 

[Symbolic  Execution  7.3.4] 
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6. 


ATTRIBUTES 


Attributes  areJh.e  characteristics  of  tools  or  “whole  APSEs”  which  the  E&V  user 
evaluates  tor  validates)  to_  make  assessments  and  comparisons.  Therefore,  the  attributes  are 
integral  to  finding  E&V  technology  since  all  E&V  technology  is  used  to  assess  tools  or  APSEs 
for  one  or  more  attributes.  This  manual  uses  a  hierarchy  of  software  attributes  derived  from 
one  presented  in  a  report  by  the  Rome  Air  Development  Center  [@RADC  1985].  The  focus  of 
the  report  is  planning  and  designing  quality  into  application  software  throughout  the  software 
life  cycle.  Nevertheless,  the  report  provides  an  excellent  framework  for  understanding  all  the 
issues  that  surround  software  quality  and  is  recommended  for  those  users  seeking  more 
details  into  this  complex  process.  The  RADC  report  is  therefore  chosen  as  a  basis  for  this 
section  of  the  E&V  Reference  Manual  even  though  the  focus  here  is  on  system  and  support 
software  and  also  on  specifying  and  assessing  attributes  for  software  that  is  already 
developed.  The  remainder  of  this  introductory  section  is  used  to  summarize  some  of  the 
issues  that  must  be  dealt  with  in  understanding  the  software  attributes.  These  are: 

•  The  attribute  hierarchy 

•  The  functional  dependence  of  attributes 

•  Guidance  in  the  selection  of  attributes 

•  Attribute  interrelationships. 

The  first  two  levels  of  the  attribute  hierarchy  are  shown  in  Table  6-1.  The  highest 
level  of  the  hierarchy  (performance,  design,  and  adaptation)  show  three  broad  categories  of 
acquisition  concerns  to  potential  users  with  regard  to  software.  The  next  level  shows  quality 
factors  which  are  user-oriented  terms,  each  representing  an  aspect  of  software  (or  tool) 
quality.  The  quality  factors  can  themselves  be  decomposed  into  various  criteria  as  shown  in 
Table  6-2.  The  criteria  are  software-oriented  terms  representing  software  characteristics. 
The  degree  to  which  these  characteristics  are  present  in  the  software  is  an  indication  of  the 
degree  of  presence  of  a  quality  factor.  The  criteria  can  support  more  than  one  of  the 
software  quality  factors  at  the  next  higher  level. 
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Table  6-1  Top  Level  Attribute  Hierarchy 
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ACQUISITION  CONCERN 

USER  CONCERN 

QUALITY  FACTOR 

HOW  WELL  DOES  IT  UTILIZE  A  RESOURCE? 

EFFICIENCY 

HOW  SECURE  IS  IT? 

INTEGRITY 

PERFORMANCE- 

HOW  WELL  DOES  IT 

WHAT  CONFIDENCE  CAN  BE  PLACED  IN 

RELIABILITY 

FUNCTION? 

WHAT  IT  DOES? 

HOW  WELL  WILL  IT  PERFORM  UNDER 

ADVERSE  CONDITIONS? 

SURVIVABILITY 

HOW  EASY  IS  IT  TO  USE? 

USABILITY 

HOW  WELL  DOES  IT  CONFORM  TO  THE 

CORRECTNESS 

DESIGN- 

REQUIREMENTS? 

HOW  WELL  IS  IT 
DESIGNED? 

HOW  EASY  IS  IT  TO  REPAIR? 

MAINTAINABILITY 

HOW  EASY  IS  IT  TO  VERIFY  ITS 

PERFORMANCE? 

VERIFIABILITY.  TESTABILITY 

HOW  EASY  IS  IT  TO  EXPAND,  CHANGE,  OR 
UPGRADE  ITS  CAPABILITY  OR  PERFORMANCE? 

EXPANDABILITY.  FLEXIBILITY 

ADAPTATION- 

HOW  EASY  IS  IT  TO  INTERFACE  WITH 

INTEROPERABILITY 

HOW  ADAPTABLE 

ANOTHER  SYSTEM? 

IS  IT? 

HOW  EASY  IS  IT  TO  CONVERT  FOR  USE  IN 
ANOTHER  APPLICATION? 

REUSABILITY 

HOW  EASY  IS  IT  TO  TRANSPORT? 

TRANSPORTABILITY 

The  quality  criteria  can  also  be  decomposed  into  metrics  which  are 
software-oriented  details  of  the  software  characteristics.  These  metrics  represent  specific 
questions,  checklists,  or  other  tests  that  are  used  to  determine  the  extent  to  which  the  tool  has 
that  characteristic.  The  metrics  must  often  be  tailored  to  the  particular  function(s)  that  the 
software  performs.  Other  metrics  do  not  deal  with  functionality,  but  with  aspects  of  the 
software  that  are  independent  of  the  function (s)  performed.  Because  of  the  possible 
functional  dependence  of  the  software  quality  metrics,  the  metrics  are  not  listed  in  this 
manual,  but  are  found  in  the  Guidebook  [@GB]. 
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Table  6-2  Complete  Attribute  Hierarchy 


ACQUISITION  CONCERN 


PERFORMANCE 

6.1 


DESIGN 

6.2 
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ADAPTATION 


CRITERION/ 

SECTION 


12  3  4 

E  I  R  S 

F  N  E  U 


ACCURACY 

ANOMALY  MANAGEMENT  (FAULT  OR  ERROR 

6.4.1 

6.4.2 

TOLERANCE.  ROBUSTNESS 

AUTONOMY 

6.4.5 

CAPACITY 

6.4.6 

COMMUMCATKJN  EFFECTTVM8S 

6.4.6 

corr 

6.4.11 

OlSTRSUTEDNESS 

6.4. 12 

MATURITY 

6.4.16 

OPERAttUTY  (COMMUMCATTVENESS) 

6.4.20 

POWER 

6.4.21 

PROCESS*#!  (EXECUTION)  EffECTIVENE83 

6.4.22 

RECONFKXJRAStJTY 

6.4.24 

REQURED  CONFIGURATION 

6.4.26 

STORAGE  EFFECTIVENESS 

6.4.31 

8YSTEM  ACCESSWLTY 

6.4.32 

TRAMNG 

6.4.36 

COMPLETENESS 

6.4.9 

CONSISTENCY 

8.4  10 

TRACEA86JTY 

6.4.36 

V1SWUTY  (TEST  AVAAAB8JTY) 

6.4.36 

APnJCATWN  SCEPeCENCE 

6.4.3 

AUOMENTAOLfTY 

6.4.4 

COMMONALITY  (OAT  A  AND  COMMUMCATION) 

8.4.7 

OOCUMENT  ACCESSWUTY 

6.4.13 

FUNCTIONAL  OVERLAP 

6.4.14 

FUNCTIONAL  SCOPE 

6.4.16 

0EN6LNJTY 

6.4.16 

REHOSTAWUTY  (NO€P€NO€NC€) 

6.4.26 

RETARGET AflUTY  (NOEPENOG^CE) 

6.4.27 

SYSTEM  CLARITY 

6.4.33 

SYSTEM  COMPATEMUTY 

6.4.34 

vsttuauty 

6.4.37 

GRANULARITY 

6.4.17 

MODULARITY 

6.4.19 

PROPRZTARY  RK1KTS 

6.4.23 

self-oescrpttveness 

0.4.29 

SS4PUCTTY 

6.4.29 

SOFTWARE  PRODUCTION  VfMCLE  (S) 

9.4.30 
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The  RADC  report  not  only  defines  the  software  quality  factors,  criteria,  and 
metrics,  it  also  provides  guidance  for  their  selection  and  use.  In  particular,  there  is  guidance 
for  selecting  attributes  based  on: 

•  System  characteristics 

•  Complementary  factors 

•  Shared  criteria 

•  Attribute  interrelationships. 

The  desired  .software  quality  factors  should  be  tailored  to  the  characteristics  of 
the  software  being  built  and  the  development  environment.  In  fact,  some  of  the  tools  selected 
for  the  development  environment  will  probably  be  based  on  the  system  characteristics.  For 
example,  if  a  tool  or  environment  is  to  have  a  long  life  cycle,  the  E&V  user  should  be 
concerned  with  expandability  and  maintainability.  Likewise,  if  the  system  being  built  is  a 
real-time  application,  the  developer  should  consider  acquiring  tools  that  can  assess  the 
efficiency  of  the  system.  Table  6-3  lists  some  application  and  environment  characteristics 
and  the  related  software  quality  factors  that  are  likely  to  be  important. 

Another  consideration  when  selecting  quality  factors  to  be  assessed  is  the  effect 
of  low  quality  levels  among  complementary  factors.  Four  factors  are  complementary  to  most 
other  factors  and  should  influence  the  selection  of  important  factors.  Table  6-4  shows  the 
complementary  factors: 

•  Reliability 

•  Correctness 

•  Maintainability 

•  Verifiability. 

Quality  levels  for  the  other  specified  factors  are  difficult  to  measure  accurately  when  there  are 
low  quality  levels  for  anv  of  the  four  complementary  factors.  For  example,  if  a  high  quality 
level  for  reliability  is  specified,  high  quality  levels  for  correctness  and  verifiability  should  also 
be  specified.  This  is  because  if  the  scores  for  reliability  are  high,  but  the  software  is  incorrect 
or  difficult  to  verify,  the  true  reliability  may  be  low  due  to  incorrect  software  or  uncertainty  in 
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Table  6-3  Examples  Of  Application/Environment  Characteristics 
and  Related  Software  Quality  Factors 


G-04765 


APPLICATION/ENVIRONMENT 

CHARACTERISTICS 

SOFTWARE  QUALITY  FACTORS 

HUMAN  LIVES  AFFECTED 

INTEGRITY 

RELIABILITY 

CORRECTNESS 

VERIFIABILITY,  TESTABILITY 
SURVIVABILITY 

LONG  LIFE  CYCLE 

MAINTAINABILITY 

EXPANDABILITY,  FLEXIBILITY 

EXPERIMENTAL  SYSTEM  OR  HIGH  RATE 

OF  CHANGE 

EXPANDABILITY,  FLEXIBILITY 

EXPERIMENTAL  TECHNOLOGY  IN 

HARDWARE  DESIGN 

TRANSPORTABILITY 

MANY  CHANGES  OVER  LIFE  CYCLE 

REUSABILITY 

EXPANDABILITY,  FLEXIBILITY 

REAL  TIME  APPLICATION 

EFFICIENCY 

RELIABILITY 

CORRECTNESS 

ON-BOARD  COMPUTER  APPLICATION 

EFFICIENCY 

RELIABILITY 

CORRECTNESS 

SURVIVABILITY 

PROCESSING  OF  CLASSIFIED  INFORMATION 

INTEGRITY 

INTERRELATED  SYSTEMS 

INTEROPERABILITY 

INTERACTIVE  SYSTEMS 

USABILITY 

BATCH  SYSTEMS 

EFFICIENCY 

RELIABILITY 

CORRECTNESS 
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Table  6-4  Complementary  Software  Quality  Factors 


G-04768 


COMPLEMENTARY 
QUALITY  FACTOR 


QUALITY 

FACTOR 

SPECIFIED 


EFFICIENCY 

■ 

■■ 

— 

■■■ 

— 

— 

— 

— 

INTEGRITY 

RELIABILITY 

□ 

■ 

□ 

SURVIVABILITY 

□ 

■ 

□ 

USABILITY 

■ 

□ 

□ 

□□ 

■■ 

■ 

— 

— 

— 

— 

CORRECTNESS 

_ 

MAINTAINABILITY 

□ 

VERIFIABILITY.  TESTABILITY 

* 

— 

on 

□□□ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

y 

EXPANDABILITY ,  FLEXIBILITY 

* 

INTEROPERABILITY 

* 

nnn 

REUSABILITY 

* 

con 

TRANSPORTABILITY 

* 

□□□ 

_ 
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verification.  The  complementary  quality  factors  show  that  the  attributes  are  not  independent. 
Anv  project,  regardless  of  the  type  of  system  or  application,  should  consider  the 
complementary  factors  in  the  quality  specifications. 

The  criteria  that  are  attributes  of  more  than  one  quality  factor  are  shared  criteria. 
For  example,  Table  6-2  shows  simplicity  is  a  criterion  for  five  of  the  factors.  The  beneficial 
effeet-of.  this  is  that  these_attributes  are  built  into  the  software  only  once;  likewise,  assessment 
of  these  attributes  need,  only  be  done  once.  Therefore,  costs  associated  with  specifying 
factors  that  share  common  criteria  are  generally  less  than  costs  associated  with  specifying 
factors  that  do  not  share  criteria. 

Assigning  more  than  one  quality  factor  to  an  APSE  component  can  have  either  a 
beneficial  or  an  adverse  effect,  depending  on  the  combination  of  factors  that  have  been 
specified.  Some  factors  have  criteria  that  conflict  with  another  factor;  and  some  have  criteria 
that  cooperate  with  another  factor.  Attribute  criteria  affecting  other  factors  are  shown  in  Table 
6-5.  Attributes  that  are  basic  criteria  of  a  factor  are  identified  with  an  “x”;  criteria  that  are  in  a 
positive  or  beneficial  relationship  with  another  factor  are  identified  with  a  and  criteria  that 
are  in  a  negative  or  adverse  relationship  with  another  factor  are  identified  with  a  For 
example,  operability  is  a  criterion  of  usability  and  is  shown  to  have  a  beneficial  relationship 
with  maintainability.  The  assertion  is  that  the  operability  of  usable  software  aids  in  software 
maintenance,  even  though  it  is  not  an  essential  characteristic  of  maintainability.  The 
implication  is  that  the  effort  to  maintain  software  will  be  less  if  usability  is  also  a  specified 
quality.  An  example  of  a  criterion  with  an  adverse  relationship  is  anomaly  management. 
Anomaly  management  is  a  criterion  of  reliability  and  is  shown  to  have  a  conflicting  relationship 
with  efficiency.  The  assertion  is  that  the  additional  code  required  to  perform  anomaly 
management  increases  the  runtime  and  requires  additional  memory,  thus  decreasing  the 
potential  efficiency.  This  implies  that  efficient  use  of  resources  will  be  more  difficult  to  achieve 
if  reliability  is  also  a  specified  quality  factor.  Possible  solutions  to  this  conflict  include: 

•  Budget  and  schedule  to  try  to  achieve  high  goals  for  both 
factors 

•  Lowering  goals  for  one  or  the  other  factor,  including  decreasing 
emphasis  on  the  specific  criterion  that  conflicts  and  increasing 
emphasis  on  those  that  do  not  conflict 
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Table  6-5  Beneficial  and  Adverse  Effects  of  Criteria  on 
Software  Quality  Factors 
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Y 

ACCURACY 

•.4.1 

- 

X 

ANOMALY  MANAGEMENT  (FAULT  OR  ERROR 
TOLERANCE.  ROBUSTNESS 

6.4.2 

- 

X 

X 

♦ 

AUTONOMY 

6.4.5 

X 

CAPACITY 

6.4.6 

X 

♦ 

COMMUMCATION  EFFECTIVENESS 

6.4.6 

X 

- 

- 

- 

COST 

6.4.11 

X 

WSTRlBUTEONESS 

6.4. 12 

- 

X 

♦ 

MATURITY 

6.4.16 

X 

OPERASCJTY  (COMMUMCATIVENESS) 

6.4.20 

- 

X 

♦ 

♦ 

POWER 

6.4.21 

X 

PROCESSING  (EXECUTION)  EFFECTIVENESS 

6.4.22 

X 

- 

- 

- 

RECONFIGURASUTY 

6.4.24 

X 

♦ 

- 

- 

- 

REQLNRED  CONFIGURATION 

6.4.26 

X 

STORAGE  EFFECTIVENESS 

6.4.31 

X 

- 

- 

SYSTEM  ACCESSWLTY 

6.4.32 

- 

X 

TRAMNG 

6.4.36 

X 

COMPLETENESS 

6.4.6 

X 

♦ 

♦ 

CONSISTENCY 

6.4.10 

X 

X 

♦ 

♦ 

♦ 

traceassjty 

6.4.36 

X 

♦ 

♦ 

♦ 

♦ 

V1SWUTY  (TEST  AVAJLASSJTY) 

6.4.36 

X 

X 

APPLICATION  NOEPENOENCE 

6.4.3 

X 

♦ 

AUGMENT  AWJTY 

6.4.4 

X 

COMMONALITY  (DATA  AND  COMMUMCATON) 

6.4.7 

- 

X 

DOCUMENT  ACCESSWLITY 

6.4. 13 

- 

♦ 

X 

FUNCTIONAL  OVERLAP 

6.4. 14 

X 

FUNCTIONAL  SCOPE 

6.4.15 

X 

GENERALITY 

6.4. 16 

- 

- 

- 

- 

X 

- 

X 

(WMOSTAWLITY  INO€PeNOeNCE) 

6.4.25 

- 

X 

X 

X 

RETARGET A84JTY  (INOEPCNOCNCE) 

6.4.27 

- 

X 

X 

X 

X 

system  CLARITY 

6.4.33 

X 

SYSTEM  COMPATTemTY 

6.4.34 

- 

X 

VIRTUALITY 

6.4.37 

- 

X 

GRANULARITY 

6.4.17 

X 

X 

X 

X 

X 

X 

MODULARITY 

64.16 

- 

X 

X 

X 

X 

X 

X 

X 

PROPWETARY  RIGHTS 

6  4  23 

X 

X 

3ELF-OESCWPT1VENESS 

6.426 

- 

X 

X 

X 

X 

X 

SAdPUCTTY 

6.4.26 

- 

X 

X 

X 

X 

X 

SOFTWARE  PRODUCTION  VfHKXEtSl 

6  .4  30 

_ 

__ 
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_ 

LEGEND 

X  «  BASIC  RELATIONSHIP 

♦  «  POSITIVE  EFFECT 

-  •  NEGATIVE  EFFECT 

BLANK  •  NONE  OR 

APPLICATION 
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•  Allocating  more  resources  to  the  host  system  (e.g.,  more 

efficient  processor  or  more  memory)  and  possibly  decreasing 
emphasis  on  high  software  quality  goals. 

Figure  6-1  shows  how  the  attributes  are  related  to  other  elements  of  the  E&V 
Reference  Manual  and  to  elements  in  the  E&V  Guidebook.  The  various  attributes  that  are 
useful  in  defining  assessment  objectives  for  APSEs  or  APSE  components  are  described  in  this 
chapter. 
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Figure  6-1  Attribute  Relationships 
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6.1  PERFORMANCE 


Description; 

Performance  quality  factors  deal  both  with  the  ability  of  the  software  to  function  and  with 
error  occurrences  that  affect  software  functioning.  Low  quality  factors  predict  poor 
software  performance.  [@RADC  1985] 


Cross  References; 


Software  Quality  Factors: 

[Efficiency 

8.1.1, 

Integrity 

6.1.2, 

Reliability 

6.1.3, 

Survivability 

6.1.4, 

Usability 

6.1.5] 
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6.1.1  Efficiency 


Description: 

Efficiency  deals  with  utilization  of  resources.  [@RADC  1985]  The  extent  to  which  a 
component  fulfills  its  purpose  using  a  minimum  of  computing  resources.  Of  course, 
many  of  the  ways  of  coding  efficiently  are  not  necessarily  efficient  in  the  sense  of  being 
cost-effective,  since  transportability,  maintainability,  etc.,  may  be  degraded  as  a  result. 
[@DACS  1979] 

CrQSsJtelerepces; 


Acquisition  Concern: 


[Performance 

6.1] 

Software-Oriented  Criteria: 

[Communication  Effectiveness 

6.4.8, 

Processing  Effectiveness 

6.4.22, 

Storage  Effectiveness 

6.4.31] 

Application/Environment  Characteristics: 

[Real  Time  Application, 

On-Board  Computer  Application, 

Batch  Systems] 

Complementary  Software  Quality  Factors: 

Cooperating  Criteria: 

Conflicting  Criteria: 

[Accuracy 

6.4.1, 

Anomaly  Management 

6.4.2, 

Commonality 

6.4.7, 

Generality 

6.4.16, 

Modularity 

6.4.19, 

Operability 

6.4.20, 

Reconfigurability 

6.4.24, 

Rehostability 

6.4.25, 

Retargetability 

6.4.27, 

Self-Descriptiveness 

6.4.28, 

Simplicity 

6.4.29, 

System  Accessibility 

6.4.32. 

Virtuality 

6.4.37] 
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6.1.2  Integrity 

Bgsfidptifljn 

Integrity  deals  with  software  failures  due  to  unauthorized  access.  [@RADC  1985]  The 
extent  to  which  access  to  a  component  or  associated  data  by  unauthorized  persons 
can  be  controlled.  [@E&V  Requirements] 

Cross  References; 


Acquisition  Concern: 
[Performance 


6.1] 


Software-Oriented  Criteria: 

[System  Accessibility  6.4.32] 


Application/Environment  Characteristics : 
[Human  Lives  Affected, 

Processing  of  Classified  Information] 


Complementary  Software  Quality  Factors: 

[Reliability  6.1.3, 

Correctness  6.2.1, 

Verifiability,  Testability  6.2.3] 


Cooperating  Criteria: 


Conflicting  Criteria: 

[Distributedness  6.4.12, 

Document  Accessibility  6.4.13, 

Generality  6.4.16, 

System  Compatibility  6.4.34] 


6-12 


E&V  Reference  Manual,  Version  1.1 


6.1.3  Reliability 


PAScr.ip.tiQn; 

Reliability  concerns  any  software  failure.  [@RADC  1985]  The  extent  to  which  a 
component  can  be  expected  to  perform  its  intended  functions  in  a  satisfactory  manner. 
[@DACS  1979] 


Cross  References: 


Acquisition  Concern: 


[Performance 

6.1] 

Software-Oriented  Criteria: 

[Accuracy 

Anomaly  Management 

Simplicity 

6.4.1, 

6.4.2, 
6.4.29] 

Application/Environment  Characteristics: 

[Human  Lives  Affected, 

Real  Time  Application, 

On-Board  Computer  Application, 

Batch  Systems] 

Complementary  Software  Quality  Factors: 
[Correctness 

Verifiability,  Testability 

6.2.1, 

6.2.3] 

Cooperating  Criteria: 

Conflicting  Criteria: 

[Generality 

6.4.16] 
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6.1.4  Survivability 


Description; 

Survivability  deals  with  the  continued  performance  of  software  (e.g.,  in  a  degraded 
mode)  even  when  a  portion  of  the  system  has  failed.  [@RADC  1985] 


Cross  References; 


Acquisition  Concern: 


[Performance 

6.1] 

Software-Oriented  Criteria: 

[Anomaly  Management 

Autonomy 

Distributedness 

Granularity 

Modular  V 

Reconfigurability 

6.4.2. 

6.4.5. 

6.4.12. 

6.4.17, 

6.4.19. 

6.4.24] 

Application/Environment  Characteristics : 

[Human  Lives  Affected. 

On-Board  Computer  Application] 

Complementary  Software  Quality  Factors: 
[Reliability 

Correctness 

Verifiability.  Testability 

6.1.3. 

6.2.1, 

6.2.3] 

Cooperating  Criteria: 

Conflicting  Criteria: 

[Generality 

6.4.16] 

6-14 


E&V  Reference  Manual,  Version  1.1 


6.1.5  Usability 

Description; 

Extent  to  which  resources  required  to  acquire,  install,  learn,  operate,  prepare  input  for, 
and  interpret  output  of  a  component  are  minimized. 

Cross  References: 


Acquisition  Concern: 

[Performance 

6.1] 

Software-Oriented  Criteria: 

[Capacity 

6.4.6, 

Cost 

6.4.11, 

Granularity 

6.4.17, 

Maturity 

6.4.18. 

Operability 

6.4.20, 

Power 

6.4.21. 

Required  Configuration 

6.4.26, 

Training 

6.4.36] 

Application/Environment  Characteristics: 
[Interactive  Systems] 


Complementary  Software  Quality  Factors: 


[Reliability 

6.1.3, 

Correctness 

6.2.1, 

Maintainability 

6.2.2, 

Verifiability,  Testability 

6.2.3] 

Cooperating  Criteria: 

[Anomaly  Management 

6.4.2] 

Conflicting  Criteria: 
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6.2  DESIGN 


Dg.sc.rietiQn,; 

Design  quality  factors  deal  mainly  with  software  failure  and  correction.  Low  quality 
levels  usually  result  in  repeating  a  portion  of  the  development  process  (e.g.,  redesign, 
recode,  reverify);  hence  the  term  design.  [@RADC  1985] 


Cross  References; 


Software  Quality  Factors: 

[Correctness  6.2.1. 

Maintainability  6.2.2, 

Verifiability.  Testability  6.2.3] 


6-16 


E&V  Reference  Manual,  Version  1.1 


6.2.1  Correctness 

Description; 

The  extent  to  which  software  design  and  implementation  conform  to  specifications  and 
standards.  Criteria  of  correctness  deal  exclusively  with  design  and  documentation 
formats.  [@RADC  1985]  Agreement  between  a  component’s  total  response  and  the 
stated  response  in  the  functional  specification  (functional  correctness),  and/or  between 
the  component  as  coded  and  the  programming  specification  (algorithmic  correctness). 
[@DACS  1979] 

Cross  References; 


Acquisition  Concern: 

[Design 

6.2] 

Software-Oriented  Criteria: 

[Completeness 

6.4.9, 

Consistency 

6.4.10, 

Traceability 

6.4.35] 

Application/Environment  Characteristics : 
[Human  Lives  Affected, 

Real  Time  Application, 

On-Board  Computer  Application, 
Batch  Systems] 


Complementary  Software  Quality  Factors: 


Cooperating  Criteria: 


Conflicting  Criteria: 
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6.2.2  Maintainability 
Description; 

The  ease  of  effort  in  locating  and  fixing  software  failures.  [@RADC  1985]  The  extent  to 
which  a  component  facilitates  updating  to  satisfy  new  requirements  or  to  correct 
deficiencies.  This  implies  that  the  component  is  understandable,  testable,  and 
modifiable.  [@DACS  1979] 

Cross  References: 


Acquisition  Concern: 


[Design 

6.2] 

Software-Oriented  Criteria: 

[Consistency 

6.4.10, 

Granularity 

6.4.17, 

Modularity 

6.4.19, 

Proprietary  Rights 

6.4.23, 

Self-Descriptiveness 

6.4.28, 

Simplicity 

6.4.29, 

Software  Production  Vehicle  (s) 

6.4.30. 

Visibility 

6.4.38] 

Application/Environment  Characteristics : 

[Long  Life  Cycle] 

Complementary  Software  Quality  Factors: 

[Reliability 

6.1.3, 

Correctness 

6.2.1] 

Cooperating  Criteria: 

[Completeness 

6.4.9. 

Document  Accessibility 

6.4.13, 

Operability 

6.4.20, 

Reconfigurability 

6.4.24, 

Traceability 

6.4.35] 

Conflicting  Criteria: 

[Communication  Effectiveness 

6.4.8, 

Processing  Effectiveness 

6.4.22. 

Storage  Effectiveness 

6.4.31] 
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6.2.3  Verifiability,  Testability 
Description: 

Software  design  characteristics  affecting  the  effort  to  verify  software  operation  and 
performance.  [@RADC  1985]  The  extent  to  which  a  component  facilitates  the 
establishment  of  verification  criteria  and  supports  evaluation  of  its  performance.  This 
implies  that  requirements  are  matched  to  specific  modules,  or  diagnostic  capabilities 
are  provided,  etc.  [@DACS  1979] 

Ctass  References; 


Acquisition  Concern: 

[Design 

6.2] 

Software-Oriented  Criteria: 

[Granularity 

6.4.17, 

Modularity 

6.4.19, 

Self-Descriptiveness 

6.4.28, 

Simplicity 

6.4.29, 

Visibility 

6.4.38] 

Application/Environment  Characteristics: 
[human  Lives  Affected] 


Complementary  Software  Quality  Factors: 


[Reliability 

6.1.3, 

Correctness 

6.2.1, 

Maintainability 

6.2.2] 

Cooperating  Criteria: 

[Consistency 

6.4.10, 

Operability 

6.4.20, 

Traceability 

6.4.35] 

Conflicting  Criteria: 


[Communication  Effectiveness 

6.4.8, 

Processing  Effectiveness 

6.4.22, 

Storage  Effectiveness 

6.4.31] 
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6.3  ADAPTATION 


Description: 

Adaptation  quality  factors  deal  mainly  with  using  software  beyond  its  original 
requirements,  such  as  extending  or  expanding  capabilities  and  adapting  for  use  in 
another  application  or  environment.  Low  quality  levels  predict  relatively  high  costs  for 
new  software  use.  [@RADC  1985] 


Cross  References: 


Software  Quality  Factors: 

[Expandability.  Flexibility 

6.3.1, 

Interoperability 

6.3.2, 

Reusability 

6.3.3, 

Transportability 

6.3.4] 

6-20 


E&V  Reference  Manual,  Version  1.1 


6.3.1  Expandability,  Flexibility 


Description: 

The  effort  in  increasing  software  capabilities  or  performance  or  to  accommodate 
changes  in  requirements.  [@RADC  1985]  The  extent  to  which  a  component  allows  new 
capabilities  to  be  added  and  existing  capabilities  to  be  easily  tailored  to  user  needs. 
[@DACS  1979] 

Cross  References: 


Acquisition  Concern: 


[Adaptation 

6.3] 

Software-Oriented  Criteria: 

[Augmentability 

6.4.4, 

Generality 

6.4.16, 

Granularity- 

6.4.17, 

Modularity 

6.4.19, 

Proprietary  Rights 

6.4.23, 

Retargetability 

6.4.27, 

Self-Descriptiveness 

6.4.28, 

Simplicity 

6.4.29, 

Software  Production  Vehicle(s) 

6.4.30, 

Virtuality 

6.4.37] 

Application/Environment  Characteristics : 

[Long  Life  Cycle, 

Experimental  System  or  High  Rate  of  Change, 

Many  Changes  over  Life  Cycle] 

Complementary  Software  Quality  Factors: 

[Reliability 

6.1.3, 

Correctness 

6.2.1, 

Maintainability 

6.2.2, 

Verifiability,  Testability 

6.2.3] 

Cooperating  Criteria: 

[Capacity 

6.4.6, 

Consistency 

6.4.10, 

Distributedness 

6.4.12, 

Traceability 

6.4.35] 

Conflicting  Criteria: 

[Reconfigurability 

6.4.24] 
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6.3.2  Interoperability 
Description; 

The  effort  in  coupling  software  of  one  system  to  software  of  one  or  more  other 
systems.  [@RADC  1985]  The  ability  of  APSEs  to  exchange  data  base  objects  and  their 
relationships  in  forms  usable  by  components  and  user  programs  without  conversion. 
Interoperability  is  measured  by  the  degree  to  which  this  exchange  can  be  accomplished 
without  conversion.  [@E&V  Requirements] 

Cross  References; 


Acquisition  Concern: 


[Adaptation 

6.3] 

Software-Oriented  Criteria: 

[Commonality 

6.4.7, 

Functional  Overlap 

6.4.14, 

Modularity 

6.4.19, 

Rehostability 

6.4.25, 

Retargetability 

6.4.27, 

System  Compatibility 

6.4.34] 

Application/Environment  Characteristics : 

[Interrelated  Systems] 

Complementary  Software  Quality  Factors: 

[Reliability 

6.1.3, 

Correctness 

6.2.1, 

Maintainability 

6.2.2, 

Verifiability,  Testability 

6.2.3] 

Cooperating  Criteria: 

[Generality 

6.4.16] 

Conflicting  Criteria: 
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6.3.3  Reusability 


Description; 

The  extent  to  which  a  component  can  be  adapted  for  use  in  another  application. 
[<®RADC  1985] 

Cross  References: 


Acquisition  Concern: 


[Adaptation 

6.3] 

Software-Oriented  Criteria: 

[Application  Independence 

6.4.3, 

Document  Accessibility 

6.4.13, 

Functional  Scope 

6.4.15, 

Generality  • 

6.4.16. 

Granularity 

6.4.17, 

Modularity 

6.4.19, 

Rehostability 

6.4.25, 

Retargetability 

6.4.27, 

Self-Descriptiveness 

6.4.28, 

Simplicity 

6.4.29, 

System  Clarity 

6.4.33] 

Application/Environment  Characteristics : 

[Many  Changes  over  Life  Cycle] 

Complementary  Software  Quality  Factors: 

[Reliability 

6.1.3, 

Correctness 

6.2.1, 

Maintainability 

6.2.2, 

Verifiability,  Testability 

6.2.3] 

Cooperating  Criteria: 

[Completeness 

6.4.9, 

Consistency 

6.4.10, 

Traceability 

6.4.35] 

Conflicting  Criteria: 

[Reconfigurability 

6.4.24] 
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6.3.4  Transportability 
Description: 

The  extent  to  which  a  component  can  be  adapted  for  use  in  another  environment  (e.g., 
different  host  or  target  hardware,  operating  system,  APSE).  [@RADC  1985]  The  ability 
of  a  component  to  be  installed  on  a  different  APSE  without  change  in  functionality. 
Transportability  is  measured  in  the  degree  to  which  this  installation  can  be 
accomplished  without  reprogramming.  [@E&V  Requirements] 

Cross  References; 


Acquisition  Concern: 

[Adaptation 

6.3] 

Software-Oriented  Criteria: 

[Modularity 

6.4.19, 

Rehostability 

6.4.25, 

Retargetability 

6.4.27, 

Self-Descriptiveness 

6.4.28] 

Application/Environment  Characteristics: 

[Experimental  Technology  in  Hardware  Design] 


Complementary  Software  Quality  Factors: 

[Reliability 

t.1.3, 

Correctness 

6.*.1, 

Maintainability 

6.2.2, 

Verifiability.  Testability 

6.2.3] 

Cooperating  Criteria: 

[Application  Independence 

6.4.3] 

Conflicting  Criteria: 

[Communication  Effectiveness 

6.4.8, 

Processing  Effectiveness 

6.4.22, 

Reconfigurability 

6.4.24, 

Storage  Effectiveness 

6.4.31] 
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6.4  SOFTWARE-ORIENTED  CRITERIA 

6.4.1  Accuracy 


Description: 

Those  characteristics  of  software  which  provide  the  required  precision  in  calculations 
and  outputs.  [@RADC  1985] 

Cross  References: 


Software  Quality  Factors: 

[Reliability  6.1.3] 

Beneficial  Quality  Factors: 

Adverse  Quality  Factors: 

[Efficiency  6.1.1] 


guidebook  References; 
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6.4.2  Anomaly  Management,  Fault  or  Error  Tolerance,  Robustness 


Description: 

Those  characteristics  of  software  which  provide  for  continuity  of  operations  under  and 
recovery  from  non-nominal  conditions.  [@RADC  1985]  The  protection  of  a  component 
from  itself,  user  errors,  and  system  errors.  The  ability  to  recover  and  provide 
meaningful  diagnostics  in  the  event  of  unforeseen  situations.  A  robust  routine  will  avoid 
failing  for  input  values  where  the  desired  output  is  well-defined,  but  the  intermediate 
results  of  a  straightforward  implementation  may  cause  the  routine  to  fail.  [@E&V 
Requirements] 


Cross  References; 


Software  Quality  Factors: 

[Reliability  6.1.3, 

Survivability  6.1.4] 


Beneficial  Quality  Factors: 
[Usability 


6.1.5] 


Adverse  Quality  Factors: 
[Efficiency 


6.1.1] 


QuidsbftolLBsfsrsD£asi 

'[Compilation  7. 1.6. 7, 

@GB:  ARTEWG  Catalogue  of  Ada  Runtime 
Implementation  Dependencies 


5.10] 


'NOTE:  Normally,  the  concern  of  evaluation  is  the  quality  of  a  tool.  However,  this 
evaluation  technique  focuses  on  the  products  of  the  tool  rather  than  the  tool  itself. 
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6.4.3  Application  Independence 

Description; 

Those  characteristics  of  software  which  determine  its  non-dependency  on  database 
system,  microcode,  computer  architecture,  and  algorithms.  [@RADC  1985] 


Cross  References; 


Software  Quality  Factors: 

[Reusability  6.3.3] 


Beneficial  Quality  Factors: 

[Transportability  6.3.4] 


Adverse  Quality  Factors: 


Guidebook  References; 
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6.4.4  Augmentabllity 


Description: 

Those  characteristics  of  software  which  provide  for  expansion  of  capability  for  functions 
and  data.  [@RADC  1985] 

Cross  References: 


Software  Quality  Factors: 

[Expandability,  Flexibility  6.3.1] 

Beneficial  Quality  Factors: 

Adverse  Quality  Factors: 


Guidebook  References: 
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6.4.5  Autonomy 

Description: 

Those  characteristics  of  software  which  determine  its  non-dependency  on  interfaces 
and  functions.  [@RADC  1985] 

Cross  References; 


Software  Quality  Factors: 

[Survivability  6.1.4] 

Beneficial  Quality  Factors: 

Adverse  Quality  Factors: 

Guidebook  References: 
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6.4.6  Capacity 


Description; 

The  upper  and  lower  limits  of  the  functions  implemented  by  a  tool.  [@E&V 
Requirements] 


Cross  References: 


Software  Quality  Factors: 

[Usability  6.1.4] 


Beneficial  Quality  Factors: 

[Expandability.  Flexibility  6.3.1] 


Adverse  Quality  Factors: 


Guidebook  References: 

[Whole  APSE  Assessment  Issues 
Compilation 
Compilation 


3.,  @GB:  APSE  Characterization  13.1: 

7.1 .6.7.  @GB:  IDA  Benchmarks  5.2; 

7.1. 6.7.  @GB:  MITRE  Benchmark 

Generator  Tool  (BGT)  5.6] 
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i 


6.4.7  Commonality  (Data  and  Communication) 


Description: 

Those  characteristics  of  software  which  provide  for  the  use  of  interface  standards  for 
protocols,  routines,  and  data  representations.  [@RADC  1985]  The  set  of  assumptions 
made  by  the  component  and  made  about  the  component  by  the  remaining  components 
and  the  system  in  which  it  appears.  Software  components  have  control,  data,  and 
service  interfaces.  Included  in  this  attribute  is  conformance  to  any  existing  pertinent 
interface  standards  such  as  the  CAIS.  [@DACS  1979] 


Cross  References: 


Software  Quality  Factors: 

[Interoperability  6.3.2] 

Beneficial  Quality  Factors: 

Adverse  Quality  Factors: 

[Efficiency  6.1.1] 
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6.4.8  Communication  Effectiveness 


Description; 

Those  characteristics  of  the  software  which  provide  for  minimum  utilization  of 
communications  resources  in  performing  functions.  [@RADC  1985] 


Cross  References; 


Software  Quality  Factors: 

[Efficiency  6.1.1] 


Beneficial  Quality  Factors: 


Adverse  Quality  Factors: 

[Maintainability  6.2.2. 

Verifiability,  Testability  6.2.3, 

Transportability  6.3.4] 
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6.4.9  Completeness 


Description: 

The  extent  to  which  a  component  provides  the  complete  set  of  operations  necessary  to 
perform  a  function.  [@E&V  Requirements] 


Cross  References: 


Software  Quality  Factors: 

[Correctness  6.2.1] 


6.2.2, 

6.3.3] 


Adverse  Quality  Factors: 


Beneficial  Quality  Factors: 
[Maintainability 
Reusability 


GuktebooK  References; 


[Compilation 

Kernel 


7. 1.6.7,  @GB:  ACVC 
•  7.2. 3.3,  @GB:  CIVC 


5.1: 

8.1] 
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6.4.10  Consistency 


Pescr.ip.tiQn; 

Those  characteristics  of  software  which  provide  for  uniform  design  and  implementation 
techniques  and  notation.  [@RADC  1985] 


Cross  References: 


Software  Quality  Factors: 

[Correctness  6.2.1. 

Maintainability  6.2.2] 


Beneficial  Quality  Factors: 

[Verifiability,  Testability  6.2.3, 

Expandability,  Flexibility  6.3.1, 

Reusability  6.3.3] 


Adverse  Quality  Factors: 


Quid.e.kQQk  References; 


6-34 


E&V  Reference  Manual,  Version  1.1 


6.4.11  Cost 

Description: 

The  cost  of  a  complete  component  or  the  costs  of  features  of  a  component.  The  cost 
of  a  component  may  vary  depending  on  delivery  with  source  code  or  object  code  only 
(for  example).  Other  cost  considerations  are  installation,  user  assistance,  and 
maintenance  support.  [@E&V  Requirements] 


Cross  References: 

Software  Quality  Factors: 

[Usability  6.1.5] 

Beneficial  Quality  Factors: 

Adverse  Quality  Factors: 

Guidebook  References; 

[Whole  APSE  Assessment  Issues  3.,  @GB:  APSE  Characterization  13.1] 
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6.4.12  Distributedness 

Description: 

Those  characteristics  of  software  which  determine  the  degree  to  which  software 
functions  are  geographically  or  logically  separated  within  the  system.  [@RADC  1985] 


Cross  References: 


Software  Quality  Factors: 

[Survivability  6.1.4] 

Beneficial  Quality  Factors: 

[Expandibility.  Flexibility  6.3.1] 

Adverse  Quality  Factors: 

[Integrity  6.1.2] 


Guidebook  References: 
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6.4.13  Document  Accessibility 

Description; 

Those  characteristics  of  software  which  provide  for  easy  access  to  software  and 
selective  use  of  its  components.  f@RADC  1985] 


Cross  References; 


Software  Quality  Factors: 

[Reusability  6.3.3] 

Beneficial  Quality  Factors: 

[Maintainability  6.2.2] 

Adverse  Quality  Factors: 

[Integrity  6.1.2] 
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6.4.14  Functional  Overlap 


Description; 

Those  characteristics  of  software  which  provide  common  functions  to  both  systems. 
[@RADC  1985] 

Cross  References; 


Software  Quality  Factors: 

[Interoperability  6.3.2] 

Beneficial  Quality  Factors: 


Adverse  Quality  Factors: 
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6.4.15  Functional  Scope 

Description; 

Those  characteristics  of  software  which  provide  commonality  of  functions  among 
applications.  [@RADC  1985] 

Cross.  References; 


Software  Quality  Factors: 

[Reusability  6.3.3] 


Beneficial  Quality  Factors: 
Adverse  Quality  Factors: 


Guidebook  Re.ferenc_es; 


6-39 


E&V  Reference  Manual,  Version  1.1 


6.4.16  Generality 


Description: 

Those  characteristics  of  software  which  provide  breadth  to  the  functions  performed  with 
respect  to  the  application.  [@RADC  1985] 


Cross  References: 


Software  Quality  Factors: 


[Expandability,  Flexibility 

6.3.1, 

Reusability 

6.3.3] 

Beneficial  Quality  Factors: 


[Interoperability 

6.3.2] 

Adverse  Quality  Factors: 

[Efficiency 

6.1.1, 

Integrity 

6.1.2, 

Reliability 

6.1.3, 

Survivability 

6.1.4] 
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6.4.17  Granularity 


Description: 

The  degree  lo  which  a  component  has  separate  limited  functions  that  are  composable, 
user  selectable,  and  communicate  through  a  common  data  base.  [@E&V 
Requirements] 


Cross  References: 


Software  Quality  Factors: 

[Survivability 

6.1.4, 

Usability 

6.1.5, 

Maintainability 

6.2.2, 

Verifiability,  Testability 

6.2.3, 

Expandability,  Flexibility 

6.3.1, 

Reusability 

6.3.3] 

Beneficial  Quality  Factors: 


Adverse  Quality  Factors: 
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6.4.18  Maturity 


Description; 

The  extent  to  which  a  component  has  been  used  in  the  development  of  deliverable 
software  by  typical  users  and  to  which  the  feedback  from  that  use  has  been  reflected 
in  modifications  to  the  component.  [@E&V  Requirements] 


Cross  References: 

Software  Quality  Factors: 

[Usability  6.1.5] 

Beneficial  Quality  Factors: 

Adverse  Quality  Factors: 

GuldoboQiLRfifor.ftnigflai 

[Whole  APSE  Assessment  Issues  3.,  @GB:  APSE  Characterization  13.1] 
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6.4.19  Modularity 


Description; 

Those  characteristics  of  software  which  provide  a  structure  of  highly  cohesive 
components  with  optimum  coupling.  [@RADC  1985]  The  extent  to  which  a  component 
is  implemented  in  a  hierarchical  structure  in  which  identifiable  functions  are  isolated  in 
separate  compilation  units. 


Cross  References: 


Software  Quality  Factors: 


[Survivability 

6.1.4. 

Maintainability 

6.2.2. 

Verifiability.  Testability 

6.2.3, 

Expandability,  Flexibility 

6.3.1. 

Interoperability 

6.3.2. 

Reusability 

6.3.3. 

Transportability 

6.3.4] 

Beneficial  Quality  Factors: 

Adverse  Quality  Factors: 

[Efficiency 

6.1.1] 

Guide&QQk-Refenencas; 
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6.4.20  Operability,  Communicativeness 


Description: 

Those  characteristics  of  software  which  determine  operations  and  procedures 
concerned  with  operation  of  software  and  which  provide  useful  inputs  and  outputs  which 
can  be  assimilated.  [@RADC  1985]  The  user/component  dialog  established  to  control 
the  execution  of  the  component.  This  is  driven  by  the  set  of  assumptions  made  by  the 
component  and  made  about  the  component  by  the  persons  who  use  it. 


Cross  References: 


Software  Quality  Factors: 
[Usability 


Beneficial  Quality  Factors: 
[Maintainability 
Verifiability,  Testability 


Adverse  Quality  Factors: 

[Efficiency 

QuidebQQfc  FteferenQSs; 

[Whole  APSE  Assessment  Issues 


6.1.5] 

6.2.2, 

6.2.3] 

6.1.1] 

3.,  @GB:  APSE  Characterization  13.1] 
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6.4.21  Power 


Description; 

The  extent  to  which  a  component  has  capabilities,  such  as  default  options  and  wild 
card  operations,  that  contribute  to  the  effectiveness  of  the  user.  [@E&V  Requirements] 


Cross  References: 


Software  Quality  Factors: 

[Usability  6.1.5] 


Beneficial  Quality  Factors: 


Adverse  Quality  Factors: 


Guidebook  Referenc.es; 


[Whole  APSE  Assessment  Issues 

3.. 

@GB:  APSE  Characterization 

13.1; 

Text 

7.1. 1.1. 

@GB:  Text  Editing  Capabilities  Checklist 

99.1; 

Preliminary  Design 

7.1. 6.4, 

@GB:  SEI  Design  Support  Equipment 

9.1; 

Detailed  Design 

7.1. 6.5, 

@GB:  SEI  Design  Support  Equipment 

9.1; 

Assembling 

7.1. 6. 6, 

@GB:  Assembling  Checklist 

6.1; 

Compilation 

7.1. 6.7, 

@GB:  Compilation  Checklist 

5.8; 

Linking/Loading 

7.1.6.13, 

@GB:  Linking/Loading  Checklist 

6.2; 

Data  Base  Management 

7.2.1. 1, 

@GB:  Data  Base  Management  Checklist 

99.2; 

File  Management 

7. 2. 1.3, 

@GB:  File  Management  Checklist 

99.7; 
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Electronic  Mail  7.2.1. 4, 


@GB:  Electronic  Mail  Checklist  99.3 

Program  Library  Management  7.2.1. 7, 

@GB:  Program  Library  Management  Checklist  5.9; 

Performance  Monitoring  7.2.1.10, 

@GB:  Performance  Monitoring  Checklist  99.5 

Configuration  Management  7. 2.2.7, 

@GB:  Configuration  Management  Capabilities  Checklist  10.1 
Configuration  Management  7.2.2.7, 

@GB:  SEI  Configuration  Management  Experiment  10.2 

Runtime  Environment  7.2.3.5, 

@GB:  ARTEWG  Runtime  Environment  Taxonomy  5.11 

Import/Export  7.2.3.6, 

@GB:  Import/Export  Capabilities  Checklist  6.3; 

Requirements  Prototyping  7.3.2.2, 

@GB:  Requirements  Prototyping  Capabilities  Checklist  99.4 

Simulation  and  Modeling  7.3.2.3, 

@GB:  Simulation  and  Modeling  Capabilities  Checklist  99.6 
Debugging  7.3.2.5, 

@GB:  Debugging  Capabilities  Checklist  6.5; 

Debugging  7.3.2.5, 

@GB;  SEI  Unit  Testing  and  Debugging  Experiment  7.2; 

Testing  7.3.2.10, 

@GB:  Testing  Capabilities  Checklist  7.1; 

Testing  7.3.2.10, 

@GB:  SEI  Unit  Testing  and  Debugging  Experiment  7.2; 

Emulation  7.3.2.13, 

@GB:  Emulation  Capabilities  Checklist  6.4; 

Timing  7.3.2.14, 

@GB:  Timing  Analysis  Capabilities  Checklist  6.6; 

Tuning  7.3.2.15, 

@GB:  Tuning  Analysis  Capabilities  Checklist  6.7; 

Real  Time  Analysis  7.3.2.17, 

@GB:  Real  Time  Analysis  Capabilities  Checklist  6.8] 
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6.4.22  Processing  (Execution)  Effectiveness 


Description: 

Those  characteristics  of  the  software  which  provide  for  minimum  utilization  of 
processing  resources  in  performing  functions.  [@RADC  1985]  The  choice  between 
alternative  algorithms  based  on  those  taking  the  least  amount  of  time.  [@DACS  1979] 


Cross  References: 


Software  Quality  Factors: 

[Efficiency  6.1.1] 


Beneficial  Quality  Factors: 


Adverse  Quality  Factors: 

[Maintainability  6.2.2. 

Verifiability.  Testability  6.2.3, 

Transportability  6.3.4] 


Guidebook  References: 


[Preliminary  Design  7.1. 6.4, 

@GB:  SEI  Design  Support  Experiment 
Detailed  Design  7. 1.6. 5, 

@GB:  SEI  Design  Support  Experiment 
•Compilation  7.1. 6.7, 

@GB:  IDA  Benchmarks 

•Compilation  7. 1.6.7, 

@GB:  ACEC 

•Compilation  7. 1.6.7, 

@GB:  PIWG  Benchmark  Tests 

•Compilation  7. 1.6.7, 


@GB:  University  of  Michigan  Benchmark  Tests 


9.1: 

9.1: 

5.2: 

5.3: 

5.4: 

5.5: 


•NOTE:  Normally,  the  concern  of  evaluation  is  the  quality  of  a  tool.  However,  this 
evaluation  technique  focuses  on  the  products  of  the  tool  rather  than  the  tool  itself. 
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’Compilation  7.1. 6.7, 

@GB:  ARTEWG  Catalogue  of  Ada  Runtime 

Implementation  Dependencies  5.10; 

Configuration  Management  7.2. 2.7, 

@GB:  SEI  Configuration  Management  Experiment  10.2; 

Debugging  7.3.2.S, 

@GB:  SEI  Unit  Testing  and  Debugging  Experiment  7.2; 

Testing  7.3.2.10, 

@GB:  SEI  Unit  Testing  and  Debugging  Experiment  7.2] 


’NOTE:  Normally,  the  concern  of  evaluation  is  the  quality  of  a  tool.  However,  this 
evaluation  technique  focuses  on  the  products  of  the  tool  rather  than  the  tool  itself. 
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6.4.23  Proprietary  Rights 
Description; 

Restrictions  on  the  release,  distribution,  or  use  of  a  component.  This  includes  so  called 
“data  rights”  restrictions.  [@E&V  Requirements] 

Cross  References: 

Software  Quality  Factors: 

[Maintainability  6.2.2, 

Expandability,  Flexibility  6.3.1] 

Beneficial  Quality  Factors: 

Adverse  Quality  Factors: 
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6.4.24  Reconfigurability 


Description; 

Those  characteristics  of  software  which  provide  for  continuity  of  system  operation  when 
one  or  more  processors,  storage  units,  or  communication  links  fails.  [@RADC  1985] 


Cross  References: 


Software  Quality  Factors: 

[Survivability  6.1.4] 


Beneficial  Quality  Factors: 

[Maintainability  6.2.2] 


Adverse  Quality  Factors: 

[Efficiency 

6.1.1, 

Expandability,  Flexibility 

6.3.1, 

Reusability 

6.3.3, 

Transportability 

6.3.4] 

Guidebook  Referanc.es; 


6-50 


E&V  Reference  Manual,  Version  1.1 


6.4.25  Rehostability 
Description: 

The  ability  of  an  APSE  component  to  be  installed  on  a  different  host  or  different 
operating  system  with  needed  reprogramming  localized  to  the  KAPSE  or  machine 
dependencies.  [@E&V  Requirements] 

Cross  References: 

Software  Quality  Factors: 

[Interoperability  6.3.2, 

Reusability  6.3.3, 

Transportability  6.3.4] 

Beneficial  Quality  Factors: 

Adverse  Quality  Factors: 

[Efficiency  6.1.1] 

Guidebook  References: 

[@GB:  Ada-Europe  Host  and  Target  Checklist  14.1, 

@GB:  Ada-Europe  Machine-Specific  Characteristics  Checklist  14.2] 
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6.4.26  Required  Configuration 
Description: 

The  amount  and  types  of  hardware  or  software  facilities  needed  for  the  operation  of  a 
component.  This  includes  primary  and  secondary  storage  and  any  other  required 
resources.  [@E&V  Requirements] 


Cross  References: 


Software  Quality  Factors: 

[Usability  6.1.5] 

Beneficial  Quality  Factors: 

Adverse  Quality  Factors: 


Guidebook  References; 

[Whole  APSE  Assessment  Issues  3.,  @GB:  APSE  Characterization  13.1] 
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6.4.27  Retargetability 


Description: 

The  ability  of  an  APSE  component  to  accomplish  its  function  with  respect  to  another 
target.  The  component  may  require  modification.  [@E&V  Requirements] 


Cross  References: 


Software  Quality  Factors: 


[Expandability,  Flexibility 

6.3.1, 

Interoperability 

6.3.2. 

Reusability 

6.3.3, 

Transportability 

6.3.4] 

Beneficial  Quality  Factors: 

Adverse  Quality  Factors: 

[Efficiency 

’  6.1.1] 

GuictebQQkJteterenw; 

[@GB:  Ada-Europe  Host  and  Target  Checklist  14.1, 

@GB:  Ada-Europe  Machine-Specific  Characteristics  Checklist  14.2: 
'Compilation  7. 1.6.7, 

@GB:  ARTEWG  Catalogue  of  Ada 

Runtime  Implementation  Dependencies  5.10] 


'NOTE:  Normally,  the  concern  of  evaluation  is  the  quality  of  a  tool.  However,  this 
evaluation  technique  focuses  on  the  products  of  the  tool  rather  than  the  tool  itself. 
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6.4.28  Self-Descriptiveness 


Description: 

Those  characteristics  of  software  which  provide  explanation  of  the  implementation  of 
functions.  [@RADC  1985]  The  technical  data,  including  on-line,  documentation,  listings, 
and  printouts,  which  serve  the  purpose  of  elaborating  the  design  or  details  of  a 
component.  [@DACS  1979] 

Cross  References; 


Software  Quality  Factors: 

[Maintainability 

6.2.2, 

Verifiability,*  Testability 

6.2.3, 

Expandability,  Flexibility 

6.3.1, 

Reusability 

6.3.3, 

Transportability 

6.3.4] 

Beneficial  Quality  Factors: 


Adverse  Quality  Factors: 

[Efficiency  6.1.1] 


QuitiebooK  References; 
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6.4.29  Simplicity 
Description: 

Those  characteristics  of  software  which  provide  for  definition  and  implementation  of 
functions  in  the  most  non-complex  and  understandable  manner.  [@RADC  1985]  A 
simple  program  style  is  one  which  makes  the  program,  as  a  whole,  easy  to  understand. 
Other  things  being  equal,  the  style  is  concise  and  straightforward.  It  makes  use  of 
process,  procedural,  and  data  abstraction,  as  appropriate,  to  present  a  clear  exposition 
of  the  intent. 


Cross  References: 


Software  Quality  Factors: 


[Reliability 

6.1.3, 

Maintainability 

6.2.2, 

Verifiability,  Testability 

6.2.3, 

Expandability,  Flexibility 

6.3.1, 

Reusability 

6.3.3] 

Beneficial  Quality  Factors: 

Adverse  Quality  Factors: 

[Efficiency 

6.1.1] 

Guidebook  References: 
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6.4.30  Software  Production  Vehicle  (s) 

Description: 

The  methodology (ies),  language(s),  and  technique(s)  used  to  produce  the  software 
related  to  a  component.  [@E&V  Requirements] 


Cross  References: 


Software  Quality  Factors: 

[Maintainability  6.2.2, 

Expandability,  Flexibility  6.3.1] 


Beneficial  Quality  Factors: 


Adverse  Quality  Factors: 


QuldebQQK  References; 
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6.4.31  Storage  Effectiveness 


Description; 

Those  characteristics  of  the  software  which  provide  for  minimum  utilization  of  storage 
resources  in  performing  functions.  [@RADC  1985]  The  choice  between  alternative 
source  code  constructions  based  on  those  taking  the  minimum  number  of  words  of 
object  code  or  in  which  the  information-packing  ...  is  high. 

[@DACS  1979] 


Cross  References: 


Software  Quality  Factors: 

[Efficiency 

6.1.1] 

Beneficial  Quality  Factors: 
Adverse  Quality  Factors: 


[Maintainability 

6.2.2, 

Verifiability.  Testability 

6.2.3, 

Transportability 

6.3.4] 

Guidebook  References; 

[Preliminary  Design 

7.1. 6.4, 

@GB:  SEI  Design  Support  Experiment 

9.1; 

Detailed  Design 

7.1. 6.5, 

@GB:  SEI  Design  Support  Experiment 

9.1: 

'Compilation 

7.1. 6.7, 

@GB:  IDA  Benchmarks 

5.2: 

'Compilation 

7.1. 6.7, 

@GB:  ACEC 

5.3: 

'Compilation 

7. 1.6.7, 

@GB:  PIWG  Benchmark  Tests 

5.4: 

'Compilation 

7.1. 6. 7, 

@GB:  ARTEWG  Catalogue  of  Ada 

Runtime  Implementation  Dependencies 

5. 1C 

'NOTE:  Normally,  the  concern  of  evaluation  is  the  quality  of  a  tool.  However,  this 
evaluation  technique  focuses  on  the  products  of  the  tool  rather  than  the  tool  itself. 


6-57 


E&V  Reference  Manual,  Version  1.1 


6.4.32  System  Accessibility 


Those  characteristics  of  software  which  provide  for  control  and  audit  of  access  to  the 
software  and  data.  [@RADC  1985] 


Software  Quality  Factors: 

[Integrity  6.1.2] 

Beneficial  Quality  Factors: 

Adverse  Quality  Factors: 

[Efficiency  6.1.1] 
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6.4.33  System  Clarity 
Description: 

Those  characteristics  of  software  which  provide  for  clear  description  of  program 
structure  in  a  non-complex  and  understandable  manner.  [@RADC  1985] 


Cross  References: 

Software  Quality  Factors: 

[Reusability  6.3.3] 

Beneficial  Quality  Factors: 

Adverse  Quality  Factors: 

Guidebook.  BafacfiDCfiSi 
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6.4.34  System  Compatibility 


Description: 

Those  characteristics  of  software  which  provide  the  hardware,  software,  and 
communication  compatibility  of  two  systems.  [@RADC  1985] 


Cross  References: 


Software  Quality  Factors: 

[Interoperability  6.3.2] 

Beneficial  Quality  Factors: 

Adverse  Quality  Factors: 

[Integrity  6.1.2] 


QuidebooK  References; 
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6.4.35  Traceability 

Description; 

Those  characteristics  of  software  which  provide  a  thread  of  origin  from  the 
implementation  to  the  requirements  with  respect  to  the  specified  development  envelope 
and  operational  environment.  [@RADC  1985] 


Cross  References; 


Software  Quality  Factors: 


[Correctness 

6.2.1] 

Beneficial  Quality  Factors: 

[Maintainability 

6.2.2, 

Verifiability,  Testability 

6.2.3, 

Expandability,  Flexibility 

6.3.1, 

Reusability 

6.3.3] 

Adverse  Quality  Factors: 

QuitiebooK  Bfiffitsufiss; 
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6.4.36  Training 


Description: 

Those  characteristics  of  software  which  provide  transition  from  current  operation  and 
provide  initial  familiarization.  [@RADC  1985]  The  extent  to  which  training  and  other  user 
help  is  available  from  the  vendor  of  a  component  or  from  the  component  itself, 
including  on-line,  documentation,  listings,  and  printouts,  which  serve  the  purpose  of 
providing  operating  instructions  for  using  the  component  to  obtain  desired  results. 
[@DACS  1979] 

Cross  References: 


Software  Quality  Factors: 

[Usability  6.1.5] 

Beneficial  Quality  Factors: 

Adverse  Quality  Factors: 


Guidebook  References: 
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6.4.37  Virtuality 


Description; 

Those  characteristics  of  software  which  present  a  system  that  does  not  require  user 
knowledge  of  the  physical,  logical,  or  topological  characteristics.  [@RADC  1985] 


Software  Quality  Factors: 

[Expandability,  Flexibility  6.3.1] 

Beneficial  Quality  Factors: 

Adverse  Quality  Factors: 

[Efficiency  6.1.1] 
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6.4.38  Visibility,  Test  Availability 
Description: 

Those  characteristics  of  software  which  provide  status  monitoring  of  the  development 
and  operation.  [@RADC  1985]  The  availability  of  tests  that  verify  the  correctness  or 
effectiveness  of  a  component  function  or  feature.  These  tests  may  also  verify  proper 
response  for  an  incorrect  input  or  technique.  [@E&V  Requirements] 


Cross  References: 


6.2.2, 

6.2.3] 


Beneficial  Quality  Factors: 


Software  Quality  Factors: 
[Maintainability 
Verifiability’  Testability 


Adverse  Quality  Factors: 
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7. 


FUNCTIONS 


This  chapter  provides  a  functional  taxonomy  used  for  characterizing  the 
functional  capabilities  of  a  tool,  toolset,  or  APSE.  The  top  two  levels  of  the  taxonomy  are  as 
follows: 


•  Transformation 

Editing 

Formatting 

On-Line  Assistance  Processing 

Sort/Merge 

Graphics  Generation 

Translation 

Synthesis 

•  Management 

Information  Management 
Project  Management 
Computer  System  Management 

•  Analysis 

Static  Analysis 
Dynamic  Analysis 
Formal  Verification 
Symbolic  Execution 
Problem  Report  Analysis 
Change  Request  Analysis. 

The  complete,  three-level,  taxonomy  may  be  seen  by  examining  the  Table  of  Contents  under 
Chapter  7. 


This  type  of  functional  classification  scheme  was  first  described  in  the  NBS 
Taxonomy  report  [(©Houghton  1983]  and  later  elaborated  in  the  SEE  Taxonomy  [@Kean 
1985].  The  E&V  Team  has  followed  the  general  scheme  suggested  in  the  above-referenced 
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documents,  but  with  some  further  elaboration  and  modifications  designed  to  make  the 
taxonomy  compatible  with  the  rest  of  the  multiple  indexing  scheme  used  in  other  parts  of  this 
manual. 


Figure  7-1  shows  how  the  functions  are  related  to  other  elements  of  the  E&V 
Reference  Manual  and  to  elements  in  the  E&V  Guidebook.  The  various  functions  that  are 
useful  in  defining  assessment  objectives  for  APSEs  or  APSE  components  are  described  in  this 
chapter.  The  relationships  between  tools  and,  functions  given  in  this  chapter  are  typical  (or 
traditional)  relationships.  The_reai_capabilities_should_be  determined  bv  the  tool  specifications, 
marketing  claims,  or  the  like. 
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7.1  TRANSFORMATION 

Description; 

Transformation  features  describe  how  the  subject  is  manipulated  to  accommodate  the 
user’s  needs.  They  describe  what  transformations  take  place  as  input  to  the  tool  is 
processed.  [@Kean  1985] 

Cross  References: 

Life  Cycle  Activities: 


Tools: 
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7.1.1  Editing 


Description; 

Selective  revision  of  computer-resident  data.  [@Kean  1985] 


Cross  References; 


Life  Cycle  Activities: 


Tools: 
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7.1. 1.1  Text 


Description; 

Editing  capabilities  provided  for  textual  data.  [@Kean  1985] 


Cross  References: 


Life  Cycle  Activities: 
[Global 


4.12.1] 


Tools: 

[Syntax-Directed  Editor  5.3.2, 

Word  Processor  5.4.2] 


Guidebook  References: 

[Power  6.4.21 , 

@GB:  Text  Editing  Capabilities  Checklist 


99.1] 


7-5 


E&V  Reference  Manual,  Version  1.1 


7.1. 1.2  Data 

Description: 

Editing  capabilities  provided  for  data  in  internal  (machine)  format. 

Cross  References; 

Life  Cycle  Activities: 


Tools: 


Guidebook  References: 
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7.1. 1.3  Graphics 

Dfisc.cipti.Qn; 

Editing  capabilities  provided  for  graphical  data. 


Cross  References: 

Life  Cycle  Activities: 
[Global 


[@Kean  1985] 


4.12.2] 
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7.1.2  Formatting 


Description: 

Arranging  data  according  to  predefined  and/or  user-defined  conventions.  [@Kean  1985] 


Cross  References: 


Life  Cycle  Activities: 


Tools: 

[Formatter  5.4.5, 

Spreadsheet  5.5.1] 
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7.1 .2.1  MIL-STD  Format 

Description; 

The  formatting  of  documents  as  specified  by  the  MIL-STD (s). 

Cross  References: 

Life  Cycle  Activities: 

[Global  4.12.2] 

Tools: 

Guidebook  References: 
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7.1. 2.2  Tabic  Of  Contents 

Description: 

The  production  of  a  table  of  contents  by  examining  the  contents  to  be  placed  within  the 
document. 

Cross  References: 

Life  Cycle  Activities: 

[Global  4.12.2] 

Tools: 
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7. 1.2.3  Predefined  and  User-Defined  Forms 


Description; 

The  ability  to  call  up  or  define  standard  formats  such  as  those  that  are  mentioned  in 
DOD-STD-2167A. 


Cross  References; 


Life  Cycle  Activities: 

[System  Requirements  Analysis/Design 
Software  Requirements  Analysis 
Preliminary  Design 
Detailed  Design 
Coding  And  CSU  Testing 
CSC  Integration  And  Testing 
CSCI  Testing 

System  Integration  And  Testing 
Change  Requirements 


4.2.1,  4.2.2,  4.2.6, 

4.3.1,  4.3.2  4.3.6, 

4.4.1  -  4.4.3,  4.4.6. 

4.5.1  -  4.5.3,  4.5.6, 

4.6.1  -  4.6.3,  4.6.6, 

4.7.1  -  4.7.3,  4.7.6, 

4.8.1  -  4.8.3,  4.8.5,  4.8.6, 

4.9.2,  4.9.5, 

4.11.2,  4.11.5] 
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7.1.3  On-Line  Assistance  Processing 

Description: 

User  interface  that  is  part  of  the  input/output  process  of  a  programming  support 
environment  (e.g.,  command  assistance,  assistance,  on-line  tutoring,  definition  assistance, 
etc.).  [@Kean  1985] 

Cross  References: 


Life  Cycle  Activities: 

[Global  4.12.2] 


Tools: 

[On-Line  Assistance  Processor  5.1.8] 
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7.1.4  Sort/Merge 


Description: 

The  process  of  arranging  data  in  a  specific  order  (e.g.,  alphabetical  order). 
1985] 


Cross  References: 


Life  Cycle  Activities: 


[Global  4.12.2] 


Tools: 

[Address  Book  5.5.3, 

Phone  Book  5.5.5, 

Dictionary  5.5.8] 


Guidebook  References: 


[@Kean 
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7.1.5  Graphics  Generation 


Description: 

The  input,  construction,  storage,  retrieval,  manipulation,  alteration,  and  analysis  of  pictorial 
data  (e.g.,  generation  of  system  architectures,  software  designs,  financial  analysis,  maps, 
graphs,  etc.).  [@Kean  1985] 

Cross  References: 


Life  Cycle  Activities: 
[Global 


4.12.2] 


Tools: 

[Graphics  Generator 


5.4.4] 
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7.1. 6.1  Systems  Requirements 


Description: 

The  processing  of  formal  systems  requirements  statements  into  an  internal  data  base 
representation  for  subsequent  use.  [@Kean  1985] 


Cross  References: 

Life  Cycle  Activities: 

[System  Requirements  Analysis/Design  4.2.2] 

Tools: 

Guidebook  References: 
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7.1. 6.2  Software  Requirements 


Description: 

The  processing  of  formal  software  requirements  statements  into  an  internal  data  base 
representation  for  subsequent  use.  [@Kean  1985] 


Cross  References: 


Life  Cycle  Activities: 

[Software  Requirements  Analysis  4.3.2] 


Tools: 
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7.1. 6.3  Requirements  To  Natural  Language 


Description: 

The  transformation  of  formal  requirements  language  constructs  into  English  language  text. 
[@Kean  1985] 

Cross  References: 


Life  Cycle  Activities: 

[System  Requirements  Analysis/Design  4.2.2, 

Software  Requirements  Analysis  4.3.2] 


Tools: 
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7.1. 6.4  Preliminary  Design 


Description: 

The  processing  of  formal  preliminary  design  statements  into  an  internal  data  base 
representation  for  subsequent  use.  [@Kean  1985] 


Cross  References: 


Life  Cycle  Activities: 
[Preliminary  Design 


4.4.2] 


Tools: 


Guidebook  References: 

[Power  6.4.21, 

@GB:  SEI  Design  Support  Experiment  9.1; 

Processing  Effectiveness  6.4.22, 

@GB:  SEI  Design  Support  Experiment  9.1; 

Storage  Effectiveness  6.4.31, 

@GB:  SEI  Design  Support  Experiment  9.1] 
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7.1. 6.5  Detailed  Design 


Description: 

The  processing  of  formal  detailed  design  statements  into  an  internal  data  base 
representation  for  subsequent  use.  [@Kean  1985] 


Cross  References: 


Life  Cycle  Activities: 
[Detailed  Design 


4.5.2] 


Tools: 


Guidebook  References: 

[Power  6.4.21, 

@GB:  SEI  Design  Support  Experiment  9.1; 

Processing  Effectiveness  6.4.22, 

@GB:  SEI  Design  Support  Experiment  9.1; 

Storage  Effectiveness  6.4.31, 

@CH:  SEI  Design  Support  Experiment  9.1] 
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7.1. 6.6  Assembling 


Description: 

Translating  a  program  expressed  in  an  assembly  language  into  object  code. 
1985] 


Cross  References: 


Life  Cycle  Activities: 


[Coding  And  CSU  Testing 

4.6.2, 

CSC  Integration  And  Testing 

4.7.2, 

CSCI  Testing 

4.8.2, 

System  Integration  And  Testing 

4.9.2, 

Operations  Testing  And  Evaluation 

4.10.2] 

Tools: 

[Assembler 

5.3.4] 

Guidebook .  FMsren&es; 

[Power  6.4.21,  @GB:  Assembling  Checklist  6.1] 


[@Kean 
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7.1. 6.7  Compilation 


Description: 

Translating  a  computer  program  expressed  in  a  procedural  or  problem-oriented  language 
into  object  code.  [@Kean  1985] 


Cross  References: 


Life  Cycle  Activities: 

[Coding  And  CSU  Testing 

4.6.2, 

CSC  Integration  And  Testing 

4.7.2, 

CSCI  Testing 

4.8.2, 

System  Integration  And  Testing 

4.9.2, 

Operational  Testing  And  Evaluation 

4.10.2] 

Tools: 


[Compiler 

5.3.3] 

Guidebook  References: 

'[Anomaly  Management 

6.4.2,  @GB:  ARTEWG  Catalogue  of  Ada 

Runtime  Implementation 

Dependencies 

5.10; 

Capacity 

6.4.6  ,  @GB:  IDA  Benchmarks 

5.2: 

Capacity 

6.4.6  ,  @GB:  MITRE  Benchmark 

Generator  Tool 

5.6; 

Completeness 

6.4.9,  @GB:  ACVC 

5.1: 

Power 

6.4.21,  @GB:  Compilation  Checklist 

5.8: 

'Processing  Effectiveness 

6.4.22,  @GB:  IDA  Benchmarks 

5.2: 

'Processing  Effectiveness 

6.4.22,  @GB:  ACEC 

5.3: 

'Processing  Effectiveness 

6.4.22,  @GB:  PIWG  Benchmark  Tests 

5.4: 

'Processing  Effectiveness 

6.4.22,  @GB:  University  of  Michigan 

Benchmark  Tests 

5.5: 

'Processing  Effectiveness 

6.4.22,  @GB:  ARTEWG  Catalogue  of  Ada 

Runtime  Implementation 
Dependencies  5.10; 


'NOTE:  Normally,  the  concern  of  evaluation  is  the  quality  of  a  tool.  However,  this 
evaluation  technique  focuses  on  the  product  of  the  tool  rather  than  the  tool  itself. 
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-  Retargetability  6.4.27,  @GB:  ARTEWG  Catalogue  of  Ada 


Runtime  Implementation 
Dependencies  5.10; 

*  Storage  Effectiveness  6.4.31,  @GB:  IDA  Benchmarks  5.2; 

'Storage  Effectiveness  6.4.31,  @GB:  ACEC  5.3; 

'Storage  Effectiveness  6.4.31,  @GB:  PIWG  Benchmark  Tests  5.4; 

'Storage  Effectiveness  6.4.31,  @GB:  ARTEWG  Catalogue  of  Ada 

Runtime  Implementation 
Dependencies  5.10] 


'NOTE:  Normally,  the  concern  of  evaluation  is  the  quality  of  a  tool.  However,  this 
evaluation  technique  focuses  on  the  product  of  the  tool  rather  than  the  tool  itself. 
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7.1. 6.8  Conversion 


Description; 


Modifying  an  existing  program  to  enable  it  to  operate  with  similar  functional  capabilities  in  a 
different  environment.  [@Kean  1985] 

Cross  References: 


Life  Cycle  Activities: 

[Coding  And  CSU  Testing 

4.6.2, 

CSC  Integration  And  Testing 

4.7.2, 

CSCI  Testing 

4.8.2, 

System  Integration  And  Testing 

4.9.2, 

Operational  Testing  And  Evaluation 

4.10.2] 

Tools: 


Guidebook  References; 
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7.1 .6.9  Macro  Expansion 


Description: 

Augmenting  instructions  in  a  source  language  with  user  defined  sequences  of  instructions 
in  the  same  source  language.  [@Kean  1985] 


Cross  References; 


Life  Cycle  Activities: 

[Coding  And  CSU  Testing 

4.6.2. 

CSC  Integration  And  Testing 

4.7.2, 

CSCI  Testing 

4.8.2. 

System  Integration  And  Testing 

4.9.2, 

Operational  Testing  And  Evaluation 

4.10.2] 

Tools: 


Guidebook  References; 
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7.1.6.10  Structure  Preprocessing 


Dtscfiption; 

Translating  a  computer  program  with  structured  constructs  into  its  equivalent  without 
structured  constructs.  [@Kean  1985] 


CtQss-Bsferences: 


Life  Cycle  Activities: 

[Coding  And  CSU  Testing 

4.6.2, 

CSC  Integration  And  Testing 

4.7.2, 

CSCI  Testing 

4.8.2, 

System  Integration  And  Testing 

4.9.2, 

Operational  Testing  And  Evaluation 

4.10.2] 

Tools: 


% 
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7.1.6.11  Body  Stub  Generation 


Description; 

The  creation  of  null  bodies  for  specifications  requiring  bodies  whose  corresponding  bodies 
have  yet  to  be  defined.  [@Kean  1985] 


Cross  References; 


Life  Cycle  Activities: 

[Coding  And  CSU  Testing  4.6.2, 

CSC  Integration  And  Testing  4.7.2] 


Tools: 
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7.1.6.12  Preamble  Generation 
DescriPtlQPi  ** 

Generating  the  preamble  for  a  main  program  that  contains  parameters.  [@Kean  1985] 

Cross  References; 


Life  Cycle  Activities: 

[Coding  And  CSU  Testing  4.6.2, 

CSC  Integration  And  Testing  4.7.2, 

CSCI  Testing  4-8-2- 

System  Integration  And  Testing  4.9.2, 

Operational  Testing  And  Evaluation  4.10.2] 


Tools: 
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7.1.6.13  Linking/Loading 

BfiSfidpiiflUi 

The  creation  of  a  load/executable  module  on  the  host  machine  from  one  or  more 
independently  translated  object  modules  or  load  modules  by  resolving  cross-references 
among  the  object  modules,  and  possibly  relocating  elements.  [@Kean  1985] 


Cross  References; 


Life  Cycle  Activities: 


[Coding  And  CSU  Testing 

4.6.2, 

CSC  Integration  And  Testing 

4.7.2, 

CSCI  Testing 

4.8.2, 

System  Integration  And  Testing 

4.9.2, 

Operational  Testing  And  Evaluation 

4.10.2] 

Tools: 

[Linker 

5.3.5, 

Loader 

5.3.6] 

GuidfifrQQR.  Bfiferfipfifiai 

[Power  6.4.21, 

@GB:  Unking/Loading  Checklist  6.2] 
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7.1.6.14  interpretation 


Description: 

The  translation  of  a  source  program  into  some  intermediate  data  structure,  then  executing 
the  algorithm  by  carrying  out  each  operation  given  in  the  intermediate  structure.  [@Kean 
1985] 

Cross  References: 


Life  Cycle  Activities: 

[Coding  And  CSU  Testing 

4.6.2, 

CSC  Integration  And  Testing 

4.7.2, 

CSCI  Testing 

4.8.2, 

System  Integration  And  Testing 

4.9.2, 

Operational  Testing  And  Evaluation 

4.10.2] 

Tools: 

[Interpreter 

5.3.7] 
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7.1.7  Synthesis 
Description: 

The  generation  of  programs  according  to  predefined  rules  from  a  program  specification  or 
intermediate  language.  [@Kean  1985] 

Cross  References: 

Life  Cycle  Activities: 

Tools: 

Guidebook  References: 
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7.1. 7.1  Design  Generation 

Description; 

The  translation  or  interpretation  used  to  construct  program  designs. 


Cross  References; 


Life  Cycle  Activities: 

[Preliminary  Design  4.4.2. 

Detailed  Design  4.5.2] 


Tools: 
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7.1. 7.2  Requirements  Reconstruction 


Description; 

The  extraction  of  software  requirements  statements  from  a  program  design  language. 


Cross  References; 


Life  Cycle  Activities: 

[Preliminary  Design 

4.4.4, 

Detailed  Design 

4.5.4. 

Coding  And  CSU  Testing 

4.6.4, 

CSC  Integration  And  Testing 

4.7.4, 

CSCI  Testing 

4.8.2] 

Tools: 
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7.1. 7.3  Program  Generation 


Description; 

The  translation  or  interpretation  used  to  construct  computer  programs  (e.g.,  language 
translator  generator,  syntax  analyzer  generator,  code  generator  generator,  environment 
definition  generator,  user  interface  generator,  etc.).  [@Kean  1985] 


Cross  References: 


Life  Cycle  Activities: 

[Coding  And  CSU  Testing 


4.6.2] 


Tools: 
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7.1. 7.4  Source  Reconstruction 


Description; 

The  extraction  of  lexical  and  structure  attributes  from  an  intermediate  language.  [@Kean 
1985] 


Cross  References; 


Life  Cycle  Activities: 

[Coding  And  CSU  Testing  4.6.4. 

CSC  Integration  And  Testing  4.7.4. 

CSCI  Testing  4.8.2] 


Tools: 
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7.1. 7.5  Decompilation 


Description; 

The  translation  of  machine  code  sequences  into  a  higher  level  procedure-oriented 
language  (i.e.,  back  into  the  same  language  from  which  the  machine  code  sequences 
were  generated).  [@Kean  1985] 

Cross  References: 


Life  Cycle  Activities: 

[Coding  And  CSU  Testing  4.6.4, 

CSC  Integration  And  Testing  4.7.4, 

CSCI  Testing  4.8.2] 


Tools: 
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7.1. 7.6  Disassembling 
Description; 

The  translation  of  machine  code  sequences  into  assembly  language  sequences.  [@Kean 
1985] 

Cross  References; 


Life  Cycle  Activities: 

[Coding  And  CSU  Testing  4.6.4, 

CSC  Integration  And  Testing  4.7.4, 

CSCI  Testing  4.8.2] 


Tools: 
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7.2  MANAGEMENT 

PfiSfidBiiaDl 

Features  that  aid  the  management  or  control  of  system/software  development.  [@Kean 
1985] 

Cross  References; 

Life  Cycle  Activities: 

Tools: 
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7.2.1  Information  Management 


Description; 

The  organization,  accessing,  modification,  dissemination,  and  processing  of  information 
that  is  associated  with  the  development  of  a  software  system.  [@Kean  1985] 


Cross  References; 


Life  Cycle  Activities: 


Tools: 
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7.2. 1.1  Data  Base  (Object)  Management 

Description; 

Managing  a  collection  of  interrelated  data  stored  together  with  controlled  redundancy  to 
serve  one  or  more  applications  and  independent  of  the  programs  using  the  data.  [@Kean 
1985] 

Cross  References; 


Life  Cycle  Activities: 
[Global 


4.12.1] 


Tools: 

[Data  Base  Manager 


5.1.9] 


Guidebook  References: 

[Power  6.4.21 , 

@GB:  Data  Base  Management  Checklist  99.2] 


7-40 


E&V  Reference  Manual,  Version  1.1 


7.2.1. 2  Documentation  Management 

Description; 

The  development  and  control  of  software  documentation.  [@Kean  1985] 


Life  Cycle  Activities: 

[Global  4.12.1] 


Tools: 

[Document  Manager  5.4.1] 
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7.2.1. 3  File  Management 

Description: 

Providing  and  controlling  access  to  files  associated  with  the  development  of  software. 
[@Kean  1985] 

Cross,  References: 


Life  Cycle  Activities: 
[Global 


4.12.1] 


Tools: 

[File  Manager 


5.1.6] 


Guidebook  References: 

[Power  6.4.21, 

@GB:  Rle  Management  Checklist  99.7] 
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7.2.1. 4  Electronic  Mail 


Description; 

The  process  of  receiving/sending  messages  from/to  other  system  users. 


Cross  Referencesi 


Life  Cycle  Activities: 
[Global 


4.12.1] 


Tools: 

[Electronic  Mail 


5.5.4] 


Guidebook  References; 

[Power  6.4.21, 

@GB:  Electronic  Mail  Checklist 


[@Kean  1985] 


99.3]. 
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7.2.1. 5  Electronic  Conferencing 
Description: 

The  concurrent  on-line  correspondence  between  two  or  more  users.  [@Kean  1985] 

Cross  References: 

Life  Cycle  Activities: 

[Global  4.12.1] 

Tools: 

[Electronic  Conferencing  5.5.6] 
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7.2. 1.6  Specification  Management 


Dfis.criptiQru 

The  control  of  requirements  specifications.  Specification  management  features  are 
somewhat  methodology  dependent  because  associated  with  them  are  requirements 
languages  with  formal  procedures  for  their  use.  [@Kean  1985] 


Cross  References: 


Life  Cycle  Activities: 

[Software  Requirements  Analysis  4.3.5, 

Preliminary  Design  4.4.5, 

Detailed  Design  4.5.5] 


Tools: 
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7.2.1. 7  Program  Library  Management 
Description: 

The  creation,  manipulation,  display,  and  deletion  of  the  various  components  of  a  program 
library.  A  program  library  is  a  repository  for  all  program  information  (e.g.,  source 
programs,  object  programs,  executable  programs,  documentation,  data,  etc.).  [@Kean 
1985] 

Cross  References; 


Life  Cycle  Activities: 

[Software  Requirements  Analysis 

4.3.2, 

Preliminary  Design 

4.4.2. 

Detailed  Design 

4.5.2, 

Coding  And  CSU  Testing 

4.6.2, 

CSC  Integration  And  Testing 

4.7.2, 

CSCI  Testing 

4.8.2, 

System  Integration  And  Testing 

4.9.2. 

Operational  Testing  And  Evaluation 

4.10.2] 

Tools: 

[Program  Library  Manager 

5.3.1] 

Guidebook  References; 

[Power  6.4.21, 

@GB:  Program  Library  Management  Checklist  5.9] 
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7.2.1. 8  Test  Data  Management 
Description; 

The  development  and  control  of  software  test  data.  [@Kean  1985] 


Cross  References: 


Life  Cycle  Activities: 

[Coding  And  CSU  Testing  4.6.3, 

CSC  Integration  And  Testing  4.7.3, 

CSCI  Testing  4.8.3] 


Tools: 
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7.2.1. 9  Evaluation  Results  Management 


Description; 

The  cataloguing  and  maintenance  of  the  results  from  the  operational  test  and  evaluation  of 
the  product.  [@Kean  1985] 

Cross  References; 


Life  Cycle  Activities: 

[Operational  Testing  And  Evaluation  4.10.5] 


Tools: 
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7.2.1.10  Performance  Monitoring 

Description; 

The  monitoring  of  the  performance  characteristics  of  the  finished  product.  [@Kean  1985] 


Cross  References: 


Life  Cycle  Activities: 


Tools: 


Guidebook  References: 

[Power  6.4.21, 

@GB:  Performance  Monitoring  Checklist  99.5] 
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7.2.2  Project  Management 
Description; 

The  management  of  a  system/software  development  project.  [@Kean  1985] 

Cross  References; 

Life  Cycle  Activities: 

Tools: 

Guidebook  References: 
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7.2.2. 1  Cost  Estimation 


Description: 

The  process  of  determining  the  amount  of  labor  necessary  for  the  completion  of  a  task, 
the  amount  and  potential  costs  of  computer  time  required,  etc.,  prior  to  and  during  a 
project’s  lifetime.  [<§>Kean  1985] 

Cross  References; 

Life  Cycle  Activities: 

[Global  4.12.1] 


Tools: 

[Cost  Estimator 


5.2.1] 
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1. 2.2.2  Quality  Specification 


Description: 

The  selection  and  prioritization  of  quality  factors  required  for  a  given  application: 
availability  of  techniques  to  apply  trade-off  analyses  (quality  factor  vs.  quality  factor,  and 
quality  factor  vs.  cost)  and  quantify  quality  requirements.  Potential  quality  factors  include: 
efficiency,  integrity,  reliability,  survivability,  usability,  correctness,  maintainability, 
verifiability/testability,  expandability/flexibility,  interoperability,  reusability,  and 
transportability.  [@Kean  1985] 

Cross  References: 


Life  Cycle  Activities: 

[Global  4.12.1] 


Tools: 

[Quality  Analyzer  5.2.2] 
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7.2.2.3  Scheduling 


Description: 

The  process  of  identifying  tasks,  products  to  be  delivered,  delivery  dates,  personnel 
needed  to  complete  tasks,  etc.  for  a  development  project.  [@Kean  1985] 

Cross  References: 


Life  Cycle  Activities: 
[Global 


4.12.1] 


Tools: 

[Scheduling 

Calendar 


5.2.3. 

5.5.7] 
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7.2.2A  Work  Breakdown  Structure 


Description; 

The  enumeration  of  all  work  activities  in  hierarchic  refinements  of  detail  that  divides  work 
to  be  done  into  short,  manageable  tasks  with  quantifiable  inputs,  outputs,  schedules,  and 
assigned  responsibilities.  [@Kean  1985] 

Cross  References;. 


Life  Cycle  Activities: 
[Global 


Tools: 

[Work  Breakdown  Structure 


4.12.1] 

5.2.4] 


E&V  Reference  Manual,  Version  1.1 


7.2.2.S  Resource  Estimation 


Description; 

The  estimation  of  resources  attributed  to  an  entity.  [@Kean  1985] 


Cross  References; 


Life  Cycle  Activities: 


[Software  Requirements  Analysis 

4.3.1. 

Preliminary  Design 

4.4.1, 

Detailed  Design 

4.5.1, 

Coding  And  CSU  Testing 

4.6.1, 

CSC  Integration  And  Testing 

4.7.1, 

CSCI  Testing 

4.8.1] 

Tools: 

[Resource  Estimator 

5.2.5] 
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7.2.2.6  Tracking 


Description: 

The  tracking  of  the  development  of  an  entity  through  the  software  life  cycle. 
1985] 


Cross  References: 


Life  Cycle  Activities: 

[Software  Requirements  Analysis 

4.3.4. 

Preliminary  Design 

4.4.4,. 

Detailed  Design 

4.5.4, 

Coding  And  CSU  Testing 

4.6.4, 

CSC  Integration  And  Testing 

4.7.4, 

CSCI  Testing 

4.8.4] 

Tools: 

[Tracking 

5.2.6] 

Guidebook  References; 


[@Kean 
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7. 2.2.7  Configuration  Management 

Description; 

The  establishment  of  baselines  for  configuration  items,  the  control  of  changes  to  these 
baselines,  and  the  control  of  releases  to  the  operational  environment.  [@Kean  1985] 


Cross  References; 


Life  Cycle  Activities: 
[Global 


4.12.5] 


Tools: 

[Configuration  Manager 


5.2.7] 


Guidebook  References: 

[Power  6.4.21 , 

@GB:  Configuration  Management  Checklist  10.1; 

Power  6.4.21, 

@GB:  SEI  Configuration  Management  Experiment  10.2; 

Processing  Effectiveness  6.4.22, 

@GB:  SEI  Configuration  Management  Experiment  10.2] 
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7.2.2.8  Quality  Assessment 


Description: 

The  use  of  field  data  to  determine  the  achieved  level  of  quality  in  deployed  software 
systems.  A  verification  that  the  specified  quality  requirements  have  been  met.  [@Kean 
1985] 

Cross  References; 


Life  Cycle  Activities: 


Tools: 

[Quality  Analyzer 


5.2.2] 


Guidebook  References; 


7-58 


E&V  Reference  Manual,  Version  1.1 


7.2.3  Computer  System  Management 

DesciiptioD,; 

The  management  of  hardware/software  architectures  to  support  the  life  cycle  software 
engineering  environment.  Such  services  include:  creating,  scheduling,  and  removing 
tasks:  switching  the  processor  among  tasks:  sending  messages  between  tasks;  providing 
direct  and  import/export  access:  managing  distributed  systems;  sending  files  from  one 
host  machine  to  another  on  a  network,  etc.  [@Kean  1985] 


Cross  References; 


Life  Cycle  Activities: 


Tools: 


Guidebook  References; 
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7.2.3. 1  Command  Language  Processing 

DfiS.Crigy.Ofl; 

The  processing  of  command  language  constructs  into  functions  performed  by  the 
operating  system.  [@Kean  1985] 

Cross  References; 


Life  Cycle  Activities: 

[Global  4.12.2] 


Tools: 

[Command  Language  Processor  5.1.1] 


Guidebook  References; 
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7.2.3.2  Input/Output  Support 
Description: 

The  services  for  accessing  standard  I/O  devices  such  as  disks,  tapes,  terminals,  and 
printers  from  within  a  program. 


Life  Cycle  Activities: 

[Global  4.12.2] 


Tools: 

[Runtime  Library  5.3.8] 
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7.2.3.3  Kernel 


Description; 

The  functions  which  provide  access  for  tools  and  application  programs  to  services  of  the 
native  operating  system. 


Cross  References; 


Life  Cycle  Activities: 
[Global 


4.12.2] 


'"ools: 

[Security  System 


5.1.3] 


Guidebook  References; 

[Completeness  6.4.9, 

<®GB:  CIVC  8.1: 

@GB:  Tool  Support  Interface  Evaluation  8.2] 
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7.2.3.4  Math/Statistics 


Ds&fiiipligni 

The  services  for  supporting  standard  math  and  statistical  operations. 


Cross  References; 


Life  Cycle  Activities: 


[Global  4.12.3] 


Tools: 

[Runtime  Library  5.3.8, 

Spreadsheet  5.5.1, 

Calculator  5.5.2] 
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7.2.3.5  Runtime  Environment 


Description: 

The  functions  that  can  be  expected  to  be  found  in  the  runtime  libraries  for  Ada 
implementations.  “It  should  be  noted  that  the  dividing  line  between  the  predefined  runtime 
support  library  on  the  one  hand  and  the  conventions  and  data  structures  of  a  compiler  on 
the  other  hand  is  not  always  obvious.  One  Ada  implementation  may  use  a  predefined 
routine  to  implement  a  particular  language  feature,  while  another  implementation  may 
realize  the  same  feature  through  conventions  for  the  executable  code.  Not  addressed 
here  are  elements  of  a  runtime  library  that  exist  because  of  special  characteristics  of  the 
underlying  computing  resource.”  [@ARTEWG  1988] 


Cross  References: 


Life  Cycle  Activities: 

[Coding  And  CSU  Testing  4.6.2, 

CSC  Integration  And  Testing  4.7.2, 

CSCI  Testing  4.8.2, 

System  Integration  And  Testing  4.9.2, 

Operational  Testing  And  Evaluation  4.10.2] 


Tools: 

[Job  Scheduler  5.1.4, 

Resource  Controller  5.1.5, 

Runtime  Library  5.3.8] 


Guidebook  References: 

[Power  6.4.21, 

@GB:  ARTEWG  Runtime  Environment  Taxonomy  5.11] 
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7.2.3.6  Import/Export 


The  services  for  communicating  objects  between  various  computer  systems  or  networks. 


Life  Cycle  Activities: 

[CSCI  Testing  4.8.2, 

System  Integration  And  Testing  4.9.2,  4.9.3, 

Operational  Testing  And  Evaluation  4.10.2,  4.10.3, 

Global  4.12.2] 


Tools: 

[Archive.  Backup,  And  Retrieval  System  5.1.2, 

Import/Export  System  5.1.7] 


[Power  6.4.21, 

@GB:  Import/Export  Checklist 


6.3] 
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7.3  ANALYSIS 


Description: 

The  features  that  provide  an  examination  of  a  substantial  whole  to  determine  both 
qualitative  and  quantitative  properties.  [@Kean  1985] 


Cross  References; 


Life  Cycle  Activities: 


Tools: 
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7.3.1  Static  Analysis 
Description: 

Static  analysis  features  specify  operations  on  the  subject  without  regard  to  the 
executability  of  the  subject.  They  describe  the  manner  in  which  the  subject  is  analyzed. 
[@Kean  1985] 

Cross  References: 


Life  Cycle  Activities: 

Tools: 

Guidebook  References: 
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7.3.1. 1  Comparison 


Description; 

The  determination  and  assessment  of  similarities  between  two  or  more  items.  [@Kean 
1985] 

Cross  ■Ref  erences; 


Life  Cycle  Activities: 
[Global 


4.12.4] 


Tools: 

[Comparator 


5.6.1] 
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7.3.1. 2  Spelling  Checking 

Description; 

The  identification  of  incorrectly  spelled  words.  [@Kean  1985] 


Cross  References; 


Life  Cycle  Activities: 
[Global 


Tools: 

[Spell  Checker 


4.12.4] 


5.4.3] 
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7.3.1. 3  Data  Flow  Analysis 

Description; 

The  analysis  of  the  formal  requirements  statements  to  determine  interface  consistency  and 
data  availability.  [@Kean  1985] 

Cross  References; 


Life  Cycle  Activities: 

[Software  Requirements  Analysis  4.3.4, 

Coding  And  CSU  Testing  4.6.4, 

CSC  Integration  And  Testing  4.7.4] 


Tools: 

[Data  Flow  Analyzer  5.6.2] 


Guidebook  References; 
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7.3.1. 4  Functional  Analysis 
Description; 

The  analysis  of  formally  stated  requirements  to  determine  their  consistency  and 
completeness.  [@Kean  1985] 

Cross  References; 


Life  Cycle  Activities: 

[Software  Requirements  Analysis  4.3.4] 


Tools: 

[Functional  Analyzer 
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7.3. 1.5  Interface  Analysis 


Description: 

The  checking  of  interfaces  between  program  elements  for  consistency  and  adherence  to 
predefined  rules  and/or  axioms.  [@Kean  1985] 


Cross  References: 


Life  Cycle  Activities: 


[Software  Requirements  Analysis 

4.3.4, 

Preliminary  Design 

4.4.4, 

Detailed  Design 

4.5.4, 

Coding  And  CSU  Testing 

4.6.4, 

CSC  Integration  And  Testing 

4.7.4] 

Tools: 

[Interface  Analyzer 
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7.3.1. 6  Traceability  Analysis 
Description; 

The  checking  for  internal  consistency  within  the  software  requirements  specifications. 
[@Kean  1985] 

Cross  References; 


Life  Cycle  Activities: 

[Software  Requirements  Analysis  4.3.4] 


Tools: 

[Traceability  Analyzer  5.6.5] 


Guidebook  References; 
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7.3.1. 7  Testability  Analysis 


Description: 

The  quantitative  measurement  of  the  extent  to  which  a  component  facilitates  the 
establishment  of  verification  criteria  and  supports  evaluation  of  its  performance. 


CrQgs.Etefcie.nces; 


Life  Cycle  Activities: 

[Software  Requirements  Analysis  4.3.4] 


Tools: 

[Testability  Analyzer 
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7.3.1. 8  Test  Condition  Analysis 

Description; 

The  analysis  of  formal  requirements  language  statements  to  determine  data  values  to  be 
examined  and  mechanisms  to  be  used  in  the  verification  of  test  results.  [@Kean  1985] 


Cross  References: 


Life  Cycle  Activities: 

[Software  Requirements  Analysis  4.3.4] 


Tools: 

[Test  Condition  Analyzer 
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7.3.1. 9  Quality  Analysis 


Description: 

The  quantitative  measurement  of  specified  quality  factors  for  use  during  the  evaluation  of 
software  products  (and  prediction  of  software  quality)  at  key  milestones  during 
development.  Factors  to  be  analyzed  include:  efficiency,  integrity,  reliability,  survivability, 
usability,  correctness,  maintainability,  verifiability/testability,  expandability/flexibility, 
interoperability,  reusability,  and  transportability.  [@Kean  1985] 


Cross  References: 


Life  Cycle  Activities: 


(Software  Requirements  Analysis 

4.3.4, 

Preliminary  Design 

4.4.4, 

Detailed  Design 

4.5.4, 

Coding  And  CSU  Testing 

4.6.4, 

CSC  Integration  And  Testing 

4.7.4, 

CSCI  Testing 

4.8.3] 

Tools: 

[Quality  Analyzer 

5.6.8] 

Guidebook  References: 
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7.3.1.10  Complexity  Measurement 

Description; 

The  determination  of  how  complicated  an  entity  (e.g..  routine,  program,  system,  etc.)  is  by 
evaluating  some  number  of  associated  characteristics.  [@Kean  1985] 

Cross  References; 


Life  Cycle  Activities: 

[Preliminary  Design 

4.4.4, 

Detailed  Design 

4.5.4, 

Coding  And  CSU  Testing 

4.6.4, 

CSC  Integration  And  Testing 

4.7.4] 

Tools: 

[Complexity  Measurer 

5.6.9] 

Guidebook  Ref erenceei 
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7.3.1.11  Correctness  Checking 
Description: 

The  determination  of  agreement  between  the  component  as  coded  and  the  programming 
specification  (algorithmic  correctness). 

Cross  References; 

Life  Cycle  Activities: 


Tools: 

[Correctness  Checker 


5.6.10] 


Guidebook  References: 
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7.3.1.12  Completeness  Checking 

Pescr.ipti.Qn:. 

The  assessment  of  whether  or  not  an  entity  has  all  its  parts  present  and  if  those  parts  are 
fully  developed.  [@Kean  1985] 

Cross  References: 


Life  Cycle  Activities: 


[Preliminary  Design 

4.4.4, 

Detailed  Design 

4.5.4, 

Coding  And  CSU  Testing 

4.6.4, 

CSC  Integration  And  Testing 

4.7.4] 

Tools: 

[Completeness  Checker 
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7.3.1.13  Consistency  Checking 


Description: 

The  determination  of  whether  or  not  an  entity  is  internally  consistent  in  the  sense  that  it  is 
consistent  with  its  specification.  [@Kean  1985] 


Cross  Reforoncas; 


Life  Cycle  Activities: 


[Preliminary  Design 

4.4.4, 

Detailed  Design 

4.5.4, 

Coding  And  CSU  Testing 

4.6.4, 

CSC  Integration  And  Testing 

4.7.4] 

Tools: 

[Consistency  Checker 

5.6.12] 

E&V  Reference  Manual,  Version  1.1 


7.3.1.14  Reusability  Analysis 


Description; 

The  quantitative  measurement  of  specified  reusability  factors  for  use  during  the  evaluation 
of  software  products  (and  prediction  of  software  reusability). 


Cross  References; 


Life  Cycle  Activities: 


Tools: 

[Reusability  Analyzer 
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7.3.1.15  Syntax  And  Semantics  Checking 

Deacriptian; 

The  detection  of  errors  in  the  syntax  and  semantics  of  a  formal  language. 

Crass. References; 


Life  Cycle  Activities: 


5.3.2, 

5.4.2, 
5.6.14] 


Tools: 

[Syntax-Directed  Editor 
Word  Processor 

Syntax  And  Semantics  Checking 
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7.3.1.16  Reachability  Analysis 

Description; 

The  detection  of  sections  of  code  that  cannot  be  executed  because  of  the  structure  of  the 
code  unit  that  contains  it. 


Cross  References: 


Life  Cycle  Activities: 

Tools: 

[Reachability  Analyzer  5.6.15] 

Guidebook  References; 
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7.3.1.17  Cross  Reference 

Description: 

The  referencing  of  entities  to  other  entities  by  logical  means.  [@Kean  1985] 


Cross  References: 


Life  Cycle  Activities: 

[Preliminary  Design 

4.4.4, 

Detailed  Design 

4.5.4, 

Coding  And  CSU  Testing 

4.6.4, 

CSC  Integration  And  Testing 

4.7.4] 

Tools: 

[Cross  Referencer 

5.6.16] 

Guidebook.  References; 
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7.3.1.18  Maintainability  Analysis 
Description; 

The  quantitative  measurement  of  specified  maintainability  factors  for  use  during  the 
evaluation  of  software  products  (and  prediction  of  software  maintainability). 

Cross  References; 


Life  Cycle  Activities: 

[Coding  And  CSU  Testing  4.6.4, 

CSC  Integration  And  Testing  4.7.4] 


Tools: 

[Maintainability  Analyzer 
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7.3.1.19  Invocation  Analysis 


Description: 

The  analysis  of  a  system/software  design  for  the  purpose  of  determining  calling 
relationships  between  elements.  [@Kean  1985] 


Cross  References: 


Life  Cycle  Activities: 

[Preliminary  Design  4.4.4, 

Detailed  Design  4.5.4, 

Coding  And  CSU  Testing  4.6.4] 


Tools: 

[Invocation  Analyzer 


5.6.18] 


Gulrieboek  References; 
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7.3.1.20  Scanning 

Description; 

The  examination  of  entities  sequentially  to  identify  key  areas  or  structure.  [@Kean  1985] 

Cross  References: 


Life  Cycle  Activities: 

[Preliminary  Design 

4.4.4, 

Detailed  Design 

4.5.4, 

Coding  And  CSU  Testing 

4.6.4, 

CSC  Integration  And  Testing 

4.7.4] 

Tools: 

[Scanner 
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7.3.1.21  Structured  Walkthrough 
Description; 

The  interactive  display  of  source  code  (or  its  design)  with  the  capability  for  branching  to 
subordinate  program  modules/units.  This  feature  automates  code  reading  such  that  a 
walkthrough  of  a  computer  program  can  be  performed  on  a  video  terminal.  [@Kean  1985] 


Cross  References: 


Life  Cycle  Activities; 


[Preliminary  Design 

4.4.4, 

Detailed  Design 

4.5.4, 

Coding  And  CSU  Testing 

4.6.4, 

CSC  Integration  And  Testing 

4.7.4] 

Tools; 

[Structured  Walkthrough  Tool 

5.6.20] 

E&V  Reference  Manual,  Version  1.1 


7.3.1.22  Auditing 

Description; 

The  conducting  of  an  examination  to  determine  whether  or  not  predefined  rules  have  been 
followed.  [@Kean  1985] 

Cross  References; 


Life  Cycle  Activities: 

[Coding  And  CSU  Testing  4.6.4. 

CSC  Integration  And  Testing  4.7.4] 


Tools: 

[Auditor 
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7.3.1.23  Error  Checking 


Description; 

The  determination  of  discrepancies,  their  importance,  and/or  their  cause.  [@Kean  1985] 


Cross  References: 


Life  Cycle  Activities: 

[Coding  And  CSU  Testing  4.6.4, 

CSC  Integration  And  Testing  4.7.4] 


Tools: 

[Error  Checker 


5.6.22] 


E&V  Reference  Manual,  Version  1.1 


7.3.1.24  Statistical  Analysis 


Dg.&griBticm: 

The  performance  of  statistical  data  collection  and  analysis.  [@Kean  1985] 


Cross  References: 


Life  Cycle  Activities: 

[Coding  And  CSU  Testing 


4.6.4] 


Tools: 

[Statistical  Analyzer 
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7.3.1.25  Statistical  Profiling 


Description; 


The  analysis  of  a  computer  program  to  determine  statement  types,  number  of  occurrences 
of  each  statement  type,  and  the  percentage  of  each  statement  type  in  relation  to  the 
complete  program.  [@Kean  1985] 


Cross  References: 

Life  Cycle  Activities: 


[Coding  And  CSU  Testing 

4.6.4, 

CSC  Integration  And  Testing 

4.7.4] 

Tools: 

[Statistical  Profiler 

5.6.24] 
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7.3.1.26  Structure  Checking 
Description: 

The  detection  of  structural  flaws  within  a  program  (e.g.,  improper  loop  nestings, 
unreferenced  labels,  unreachable  statements,  and  statements  with  no  successors).  Flaws 
detected  by  this  function  are  not  illegal  or  erroneous,  but  do  constitute  bad  style.  [@Kean 
1985] 

Cross  References: 


Life  Cycle  Activities: 

[Coding  And  CSU  Testing  4.6.4, 

CSC  Integration  And  Testing  4.7.4] 


Tools: 

[Structure  Checker 
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7.3.1.27  Type  Analysis 


Description: 

The  evaluation  of  whether  or  not  the  domain  of  values  attributed  to  an  entity  are  properly 
and  consistently  defined.  [@Kean  1985] 


Cross  References: 


Life  Cycle  Activities: 

[Coding  And  CSU  Testing  4.6.4, 

CSC  Integration  And  Testing  4.7.4] 


Tools: 

[Type  Analyzer 
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7.3.1.28  Units  Analysis 


Description: 

The  determination  of  whether  or  not  the  units  or  physical  dimensions  attributed  to  an  entity 
are  properly  defined  and  consistently  used.  [@Kean  1985] 


Cross  References: 


Life  Cycle  Activities: 

[Coding  And  CSU  Testing  4.6.4, 

CSC  Integration  And  Testing  4.7.4] 


Tools: 

[Units  Analyzer  5.6.27] 


QuidfibooKJelerenfigs: 
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7.3.1.29  I/O  Specification  Analysis 


Description; 

The  analysis  of  the  input  and  output  specifications  in  a  program  usually  for  the  generation 
of  test  data.  [@Kean  1985] 

Cross  References: 


Life  Cycle  Activities: 

[Coding  And  CSU  Testing  4.6.4, 

CSC  Integration  And  Testing  4.7.4] 


Tools: 

[I/O  Specification  Analyzer  5.6.28] 
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7.3.1.30  Sizing  Analysis 


Description; 

The  quantitative  measurement  of  the  maximum  amount  of  primary  (and  secondary) 
storage  required  to  run  a  program. 


Cross  References; 


Life  Cycle  Activities: 

[Coding  And  CSU  Testing  4.6.4, 

CSC  Integration  And  Testing  4.7.4] 


Tools: 

[Sizing  Analyzer 


5.6.29] 


Guidebook  References; 
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7.3.2  Dynamic  Analysis 

Description; 

Dynamic  analysis  features  specify  operations  that  are  determined  during  or  after  execution 
takes  place.  Dynamic  analysis  features  differ  from  those  classified  as  static  by  virtue  of 
the  fact  that  they  require  some  form  of  symbolic  or  machine  execution.  They  describe  the 
techniques  used  by  the  tool  to  derive  meaningful  information  about  a  program’s  execution 
behavior.  [@Kean  1985] 


Cross  References; 

Life  Cycle  Activities: 

Tools: 
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7.3.2.1  Requirements  Simulation 

Description: 

The  execution  of  code  enhanced  requirements  statements  to  examine  functional  interfaces 
and  performance.  [@Kean  1985] 

Cross  References: 


Life  Cycle  Activities: 


[System  Concepts 

4.1.2, 

System  Requirements  Analysis/Design 

4.2.4, 

Software  Requirements  Analysis 

4.3.4] 

Tools: 

[Requirements  Simulator 

5.7.1] 
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7.3.2.2  Requirements  Prototyping 


Description: 

The  rapid  construction  of  critical  functions  of  a  system  early  in  the  life  cycle  for  the 
purpose  of  understanding  the  requirements.  During  this  activity,  these  results  are  intended 
to  be  thrown  away.  [@Kean  1985] 


Cross  References: 


Life  Cycle  Activities: 

[System  Concepts  4.1.2, 

System  Requirements  Analysis/Design  4.2.2, 

Software  Requirements  Analysis  4.3.2] 


Tools: 

[Requirements  Prototype  5.7.2] 


Guidebook  References: 

[Power  6.4.21, 

@GB:  Requirements  Prototyping  Checklist  99.4] 
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7.3.2.3  Simulation  And  Modeling 


Description; 

The  representation  of  selected  characteristics  of  the  behavior  of  one  physical  or  abstract 
system  by  another  system  (e.g,,  the  representation  of  physical  phenomena  by  means  of 
operations  performed  by  a  computer  system,  the  representation  of  operations  of  a 
computer  system  by  those  of  another  computer  system,  etc.).  [@Kean  1985] 


Cross  References: 


Life  Cycle  Activities: 

[System  Concepts  4.1.2, 

Preliminary  Design  4.4.4, 

Detailed  Design  4.5.4] 


Tools: 

[Simulation  And  Modeling  Tools 


5.7.3] 


Guidebook  References: 

[Power  6.4.21, 

@GB:  Simulation  And  Modeling  Checklist  99.6] 
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7.3.2.4  Design  Prototyping 


Description: 

The  rapid  construction  of  critical  functions  of  a  system  early  in  the  life  cycle  for  the 
purpose  of  understanding  the  requirements.  During  this  activity,  the  results  will  be  retained 
for  the  purpose  of  incrementally  developing  the  product.  [@Kean  1985] 


Cress.  References; 


Life  Cycle  Activities: 

[Preliminary  Design  4.4.2, 

Detailed  Design  4.5.2] 


Tools: 

[Design  Prototype 
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7. 3.2.5  Debugging 

PesgrlptiQQ; 

The  process  of  locating,  analyzing,  and  correcting  suspected  faults  in  a  program.  [@Kean 
1985] 

Cross  References: 


Life  Cycle  Activities: 


[Coding  And  CSU  Testing 

4.6.4, 

CSC  Integration  And  Testing 

4.7.4, 

CSCI  Testing 

4.8.4] 

Tools: 

[Debugger 

QuWebook  Bfitetsnfifiai 

5.7.5] 

[Power  6.4.21, 

@GB:  Debugging  Checklist 

Power  6.4.21, 

6.5; 

@GB:  SEI  Unit  Testing  And  Debugging  Experiment 
Processing  Effectiveness  6.4.22, 

7.2; 

@GB:  SEI  Unit  Testing  And  Debugging  Experiment 

7.2] 
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7.3.2.6  Executable  Assertion  Checking 


Description: 

The  checking  of  user-embedded  statements  that  assert  relationships  between  elements  of 
a  program.  An  assertion  is  a  logical  expression  that  specifies  a  condition  or  relation 
among  the  program  variables.  Checking  may  be  performed  with  symbolic  or  run-time 
data.  [@Kean  1985] 


Cross  Beferongfis; 


Life  Cycle  Activities: 

[Coding  And  CSU  Testing  4.6.4, 

CSC  Integration  And  Testing  4.7.4] 


Tools: 

[Executable  Assertion  Checker  5.7.6] 


] 
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7.3.2.7  Constraint  Evaluation  (Contention) 

Pescriptiom 

The  generation  and/or  solution  of  path  input  or  output  constraints  for  determining  test  input 
or  proving  programs  correct.  [@Kean  1985] 


Cross  References: 


Life  Cycle  Activities: 

[Coding  And  CSU  Testing  4.6.4. 

CSC  Integration  And  Testing  4.7.4] 


Tools: 

[Constraint  Evaluator  5.7.7] 


Guidebook  References; 
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7.3.2.8  Coverage/Frequency  Analysis 


Description: 

The  determination  and  assessment  of  measures  associated  with  the  invocation  of  program 
structural  elements  to  determine  the  adequacy  of  a  test  run.  Coverage  analysis  is  useful 
when  attempting  to  execute  each  statement,  branch,  path,  or  program.  [@Kean  1985] 


Cross  References: 


Life  Cycle  Activities: 


[Coding  And  CSU  Testing 

4.6.4, 

CSC  Integration  And  Testing 

4.7.4, 

CSCI  Testing 

4.8.3] 

Tools: 

[Coverage/Frequency  Analyzer 
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7.3.2.S  Mutation  Analysis 


Description: 

The  application  of  test  data  to  a  program  and  its  “mutants”  (i.e.,  programs  that  contain 
one  or  more  likely  errors)  in  order  to  determine  test  data  adequacy.  [@Kean  1985] 


Cross  References: 


Life  Cycle  Activities: 

[Coding  And  CSU  Testing  4.6.4, 

CSC  Integration  And  Testing  4.7.4] 


Tools: 

[Mutation  Analyzer  5.7.9] 
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7.3.2.10  Testing 
Description: 

The  identification  of  necessary  test  cases  to  accomplish  testing  objectives,  the  generation 
of  actual  test  data  for  each  test  case  to  support  both  structural  and  functional  testing,  the 
execution  of  the  test  cases,  and  the  analysis  of  the  test  results. 

Cross  References: 


Life  Cycle  Activities: 


[Coding  And  CSU  Testing 

4.6.4, 

CSC  Integration  And  Testing 

4.7.4, 

CSCI  Testing 

4.8.3] 

Tools: 

[Testing  Analyzer 

5.7.10] 

Guidebook  References; 

[Power  6.4.21, 

@GB:  -T esting  Checklist  7.1; 

Power  6.4.21 , 

@GB:  SEI  Unit  Testing  And  Debugging  Experiment  7.2: 

Processing  Effectiveness  6.4.22, 

@GB:  SEI  Unit  Testing  And  Debugging  Experiment  7.2] 
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7.3.2.11  Regression  Testing 


Description; 

The  rerunning  of  test  cases  which  a  program  has  previously  executed  correctly  in  order  to 
detect  errors  spawned  by  changes  or  corrections  made  during  software  development  and 
maintenance.  [@Kean  1985] 


Cross  References; 


Life  Cycle  Activities: 

[Coding  And  CSU  Testing  4.6.4, 

CSC  Integration  And  Testing  4.7.4, 

CSCI  Testing  4.8.3] 


Tools: 

[Regression  T esting  Analyzer  5. 7. 1 1  ] 


QuidebooK  Rs-fereacesi 
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7.3.2.12  Resource  Utilization 


Description: 

The  analysis  of  resource  utilization  associated  with  system  hardware  or  software.  [@Kean 
1985] 


Cross  References: 


Life  Cycle  Activities: 


[Coding  And  CSU  Testing 

4.6.4, 

CSC  Integration  And  Testing 

4.7.4, 

CSCI  Testing 

4.8.3] 

Tools: 

[Resource  Utilization  Analyzer 
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7.3.2.13  Emulation 


Description;. 

The  imitation  of  all  or  part  of  one  computer  system  by  another,  primarily  by  hardware,  so 
that  the  imitating  computer  system  accepts  the  same  data,  executes  the  same  programs, 
and  achieves  the  same  results  as  the  imitated  system.  [@Kean  1985] 


Cross  References; 


Life  Cycle  Activities: 

[System  Concepts 

4.1.2, 

Coding  And  CSU  Testing 

4.6.4, 

CSC  Integration  And  Testing 

4.7.4, 

CSCI  Testing 

4.8.3] 

Tools: 

[Emulator 

5.7.13] 

Guidebook  .References; 

[Power  6.4.21 , 

@GB:  Emulation  Checklist 


6.4] 
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7.3.2.14  Timing 


Description; 

The  reporting  of  actual  CPU,  wail-clock,  or  other  times  associated  with  parts  of  a  program. 
[@Kean  1985] 

Cross  References; 


Life  Cycle  Activities: 

[Coding  And  CSU  Testing  4.6.4, 

CSC  Integration  And  Testing  4.7.4, 

CSCI  Testing  4.8.3] 


Tools: 

[Timing  Analyzer  5.7.14] 


Guidebook  References: 

[Power  6.4.21, 

@GB:  Timing  Checklist  6.6] 
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7.3.2.15  Tuning 


Description: 

The  activity  of  optimizing  parts  of  a  program  which  account  for  significant  amounts  of 
execution  time.  The  optimization  follows  the  determination  of  a  “performance  profile” 
which  is  the  identification  of  the  parts  of  a  program  which  use  the  most  execution  time. 


Cross  References: 


Life  Cycle  Activities: 


[Coding  And  CSU  Testing 

4.6.4, 

CSC  Integration  And  Testing 

4.7.4, 

CSCI  Testing 

4.8.3] 

Tools: 

[Tuning  Analyzer 

5.7.15] 

Quidebook-Beferences; 

[Power  6.4.21, 

@GB:  Tuning  Checklist  6.7] 
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7.3.2.16  Reliability  Analysis 
Description; 

The  determination  of  the  ability  of  an  item  to  perform  a  required  function  under  stated 
conditions  for  a  stated  period  of  time.  [@Kean  1985] 

Cross-Referencesi 


Life  Cycle  Activities: 

[Coding  And  CSU  Testing  4.6.4, 

CSC  Integration  And  Testing  4.7.4, 

CSCI  Testing  4.8.3] 


Tools: 

[Reliability  Analyzer  5.7.16] 


Quideboek  .References; 
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7.3.2.17  Real  Time  Analysis 
Description: 

The  determination  of  the  behavior  of  a  program  which  must  recognize,  interpret,  and 
respond  to  external,  asynchronous  events  at  speeds  which  allow  the  system  to  handle  all 
such  events  in  a  “reasonable”  amount  of  time. 


Life  Cycle  Activities: 


[Coding  And  CSU  Testing 

4.6.4, 

CSC  Integration  And  Testing 

4.7.4, 

CSCI  Testing 

4.8.3] 

Tools: 

[Real  Time  Analyzer 

5.7.17] 

[Power  6.4.21, 

@GB:  Real  Time  Analysis  Checklist 


6.8] 
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7.3.3  Formal  Verification 

Description: 

The  use  of  rigorous  mathematical  techniques  to  prove  the  consistency  between  an 
algorithmic  solution  and  a  rigorous,  complete  specification  of  the  intent  of  the  solution. 
[@Kean  1985] 

Cress  rtetereocss; 


Life  Cycle  Activities: 

[Preliminary  Design  4.4.3, 

Detailed  Design  4.5.3, 

Coding  And  CSU  Testing  4.6.3] 


Tools: 

[Formal  Verification  System  5.7.18] 
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7.3.4  Symbolic  Execution 

BftSfidpilflm 

The  reconstruction  of  the  logic  and  computations  along  a  program  path  by  executing  the 
path  with  symbolic  rather  than  actual  values  of  data.  [@Kean  1985] 


Cross  References: 


Life  Cycle  Activities: 

[Detailed  Design  4.5.4, 

Coding  And  CSU  Testing  4.6.4, 

CSC  Integration  And  Testing  4.7.4] 


Tools: 

[Symbolic  Execution  System  5.7.19] 
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7.3.5  Problem  Report  Analysis 


Description: 

The  analysis  of  problem  reports  for  the  purpose  of  determining  the  validity  of  the  reported 
problem  and  a  corrective  action.  [@Kean  1985] 


Cross  References: 


Life  Cycle  Activities: 

[CSC  Integration  And  Testing  4.7.4, 

CSCI  Testing  4.8.3, 

System  Integration  And  Testing  4.9.3, 

Operational  Testing  And  Evaluation  4.10.3] 


Tools: 

[Problem  Report  Analyzer  5.2.8] 
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7.3.6  Change  Request  Analysis 
Description; 

The  analysis  of  change  requests  to  determine  the  necessity  of  the  change, 
technical/economic  impacts,  and  approach  to  accomplishing  the  change.  [@Kean  1985] 

Cross  References; 


Life  Cycle  Activities: 


Tools: 

[Change  Request  Analyzer 


5.2.9] 


Guidebook  References; 
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7.3.6.1  Change  Impact  Analysis 

Description: 

The  ability  to  determine,  for  a  proposed  support/enhancement  operation,  the  impact  of 
proposed  changes  to  the  software  system.  For  example,  changes  could  be  specified  at 
the  requirements,  design,  or  code  levels  and,  utilizing  the  traceability  mechanisms  which 
link  elements  of  various  life  cycle  activities,  the  impact(s)  of  the  changes  could  be 
identified.  Impact(s)  would  include  those  to  requirements,  design,  code,  test  cases/test 
data,  and  associated  documentation.  [@Kean  1985] 


Cross  References: 


Life  Cycle  Activities: 
[Change  Requirements 


4.11.2] 
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APPENDIX  B 
GLOSSARY 


B.1  ACRONYMS  AND  ABBREVIATIONS 


This  section  provides  a  list  of  all  acronyms  and  abbreviations  used  in  this 
manual,  with  the  associated  meanings. 


APSE 

Ada  Programming  Support  Environment 

ARTEWG 

Ada  Runtime  Environments  Working  Group  (SIGAda) 

CaIS 

Common  APSE  Interface  Set 

CBD 

Commerce  Business  Daily 

CIDS 

Critical  Item  Development  Specification 

CRISD 

Computer  Resources  Integrated  Support  Document 

CSC 

Computer  Software  Component 

CSCI 

Computer  Software  Configuration  Item 

CSOM 

Computer  System  Operator’s  Manual 

CSU 

Computer  Software  Unit 

DACS 

Data  and  Analysis  Center  for  Software 

DoD 

Department  of  Defense 

ECP 

Engineering  Change  Proposal 

E&V 

Evaluation  and  Validation 

FSM 

Firmware  Support  Manual 

GB 

E&V  Guidebook 

HWCI 

Hardware  Configuration  Item 

IDD 

Interface  Design  Document 

IEEE 

Institute  of  Electrical  and  Electronics  Engineers 

IPSE 

Integrated  Project  Support  Environment 

IRS 

Interface  Requirements  Specification 

MCCS 

Mission-Critical  Computer  System 

PIDS 

Prime  Item  Development  Specification 

PIWG 

Performance  Issues  Working  Group  (SIGAda) 

PSE 

Programming  Support  Environment 

RADC 

Rome  Air  Development  Center 

RFP 

Request  for  Proposal 

RM 

E&V  Reference  Manual 

SCN 

Specification  Change  Notice 

SDD 

Software  Design  Document 

SDE 

Software  Development  Environment 
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SDF  Software  Development  File 

SDP  Software  Development  Plan 

SEE  Software  Engineering  Environment 

SIGAda  Special  Interest  Group  for  Ada  of  the  Association  for  Computing 
Machinery  (ACM) 

SOW  Statement  of  Work 

SPM  Software  Programmer's  Manual 

SPS  Software  Product  Specification 

SRS  Software  Requirements  Specification 

SSDD  System/Segment  Design  Document 

SSS  System/Segment  Specification 

STD  Software  Test  Description 

STP  Software  Test  Plan 

STR  Software  Test  Report 

SUM  Software  User’s  Manual 

VDD  Version  Description  Document 

WBS  Work  Breakdown  Structure 
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APPENDIX  C 
FORMAL  GRAMMAR 


This  appendix  specifies  sections  of  the  Reference  Manual  and  Guidebook  (Refer¬ 
ence  System)  as  a  formal  grammar.  The  sections  include  chapters  four  through  seven  of  the 
Reference  Manual  (RM).  all  explicit  references,  the  table  of  contents,  the  composite  index, 
and  the  reference  appendix.  The  specification  is  presented  as  a  partitioned  grammar  for  con¬ 
venience. 


(The  grammar  is  presented  in  a  modified  Backus-Naur  form.  Brackets  represent 
optionality  when  alone,  and  may  be  marked  by  an  asterisk  to  denote  0-N  instances  of  the 
production,  or  by  a  sharp  to  denote  1-N  instances.  Angle  brackets  denote  comments  in 
place  of  productions  which  are  too  elaborate  to  express  here.  All  terminals  of  the  grammar 
are  expressed  as  quoted  literals,  or  composite  literals  based  on  characters  and  character 
strings.) 


C.1  FORMAL  REFERENCES 

Throughout  the  Reference  System,  whenever  formal  references  are  made,  a  sin¬ 
gle  consistent  btsi  of  grammar  <uies  are  used.  This  includes  reference  from  one  volume  to  the 
other,  reference  from  one  section  in  a  volume  to  another  section  in  the  same  document,  and 
reference  to  documents  outside  the  Reference  System. 


referencejist 

references 

reference 

phrase 


“[”  references  references  ]*  “]” 

reference  [  reference  ]’ 

phrase  [  phrase  ]  [  designatorjist  ] 
I  [  phrase  ]  designatorjist 
<text  lacking  special  characters> 
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designatorjist 

designator 

leads 

digits 

caps 


designator  [  designator  ]* 
leads  I  leads  [  digits  ]* 
digits  I  caps 
('O'- 9’) 

(’A’-’Z’) 


C.2  INDEXES 

The  index  format  of  the  Reference  System  builds  on  the  reference  specifications 
above,  and  is  derived  from  MIL-STD-1815A. 


index 

entry 

topic 

redirector 

phrasejist 


=  [  entry  ]# 

=  topic  [  redirector  ]  [  reference Jist  ] 

*  phrase  [  “(”  designatorjist  “)”  ] 

=  “["  "see"  [  “also"  ]  phrasejist  “]" 
;=  phrase  [  phrase  ]* 


C.3  FORMAL  CHAPTERS 

Those  chapters  of  the  RM  which  are  derived  from  the  classification  schema  are 
formally  defined  here. 

C.3.1  Chapter  Components 

The  following  rules  define  the  components  which  are  used  to  compose  formal 
chapter  entries. 

descriptor  ::=  designator  phrase  [description] 

description  ::=  “Description:”  text 
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text  ::=  <  prose  text  > 

functions  ::=  “Functions:”  reference Jist 

products  ::=  “Products:"  reference-list 

gbreferences  ::=  “Guidebook:"  reference  Jist 
lifecycleactivities  ::=  “Life  Cycle  Activities:”  reference  Jist 
tools  ::=  “Tools:”  reference  Jist 

quality Jactors  ::=  "Software  Quality  Factors:” 
referencejist 

acquisitionconcern  ::=  “Acquisition  Concern:” 

referencejist 

software_criteria  ::=  “Software-Oriented  Criteria:” 

referencejist 

characteristics  ::=  “Application/Environment  Characteristics:” 
referencejist 

complementary  factors  ::=  “Complementary  Software  Quality  Factors:” 

referencejist 

cooperating_criteria  ::=  “Cooperating  Criteria:” 

referencejist 

conflicting  criteria  ::=  “Conflicting  Criteria:” 

referencejist 

beneficial  factors  ::=  “Beneficial  Quality  Factors:” 

referencejist 

adverse  Jactors  ::=  "Adverse  Quality  Factors:” 

reference  list 
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C.3.2  Chapter  Entries 

Each  numbered  section  of  the  formal  chapters  follows  a  specific  grammar  rule. 
The  following  rules  define  the  format  of  each  class  of  chapter  entries. 


life_cycle_activity 

management 

engineering 

testing 

evaluation 

configuration 

transition 

apse 

toolset 

tool 

attribute 

concern 

factor 

criterion 

function 


::=  descriptor  management  engineering  testing  evaluation 
configuration  transition 

descriptor  products  functions 

descriptor  products  functions 

descriptor  products  functions 

descriptor  products  functions 

descriptor  products  functions 

descriptor  products  functions 

descriptor  [  toolset  ]’ 

descriptor  [  tool  ]* 

descriptor  functions 

descriptor  [  concern  ]*  [factor  ]* 

[  criterion  ]* 

descriptor  quality  factors 

descriptor  acquisition_concern 
software_criteria  characteristics 
compiementaryjactors  cooperating_criteria 
conflicting_criteria 
descriptor  qualityjactors 
beneficialjactors  adversejactors 
gb_references 

descriptor  life_cycle_activities  tools 
gb_references 
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C.3.3  Formal  Chapter  Ordering 

The  formal  portion  of  the  RM  is  found  in  chapters  four  through  seven.  Each  of  the 
classes  of  chapter  entries  is  found  in  a  distinct  chapter. 

formal_chapters  ::=  [  life_cycle_activity  ]* 

[  apse  ] 

[  attribute  ]  * 

[  function  ]* 

C.4  TABLE  OF  CONTENTS 

The  table  of  contents  shares  some  features  with  the  rest  of  the  formal  aspects  of 

the  RMGB. 

tab!e_of_contents 
chapter  ::= 

designator_page 
index  ::= 

C.5  CITATIONS 

The  citations  are  found  in  Appendix  A,  and  have  a  formal  structure  as  defined  in 
the  following  grammar.  The  (semantic)  form  of  citation  text  is  taken  from  the  standard  for 
IEEE  Software  Magazine. 


citations 

::=  [  citation  ]* 

citations 

::=  key  body 

';ey 

-4-^ 

X 

s 

II 

body 

::=  text  |  key 

::=  [  chapter  ]*  index 
designator  phrase  designator_page 
::=  designator  digits 

“Index”  digits 
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INDEX 


Accuracy 

(6.4.1) 

Activities 

[Life  Cycle  Activities 

4.] 

Adaptation 

(6.3) 

[Adaptation  Attributes  (of  Whole  APSEs) 

3.3.3] 

Address  Book 

(5.5.3) 

Analysis 

[Functions: 

7,3. 

Change  Impact  Analysis 

7.3. 6.1, 

Change  Request  Analysis 

7.3.6. 

Coverage/Frequency  Analysis 

7. 3. 2. 8, 

Data  Flow  Analysis 

7.3. 1.3, 

Dynamic  Analysis 

7.3.2, 

Functional  Analysis 

7.3.1. 4, 

Interface  Analysis 

7.3.1. 5, 

Invocation  Analysis 

7.3.1.19, 

I/O  Specification  Analysis 

7.3.1.29, 

Maintainability  Analysis 

7.3.1.18, 

Mutation  Analysis 

7.3. 2.9, 

Problem  Report  Analysis 

7.3.5, 

Quality  Analysis 

7.3.1. 9, 

Reachability  Analysis 

7.3.1.16, 

Real  Time  Analysis 

7.3.2.17, 

Reliability  Analysis 

7.3.2.16, 

Reusability  Analysis 

7.3.1.14, 

Sizing  Analysis 

7.3.1.30, 

Static  Analysis 

7.3.1, 

Statistical  Analysis 

7.3.1.24, 

Test  Condition  Analysis 

7.3.1. 8, 

Testability  Analysis 

7.3.1. 7, 

Traceability  Analysis 

7.3.1. 6, 

Type  Analysis 

7.3.1.27, 

Units  Analysis 

7.3.1.28] 

Anomaly  Management 

(6.4.2) 

Application  Independence 

(6.4.3) 

APSE 

[APSE  Tool  Categories 

5.; 

Life  Cycle  Activities 

4.; 

Whole  APSE  Issues 

3.] 
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Archive 

[Archive.  Backup,  and  Retrieval  System 

Assemble 

[Assembler 

Assembling 

Assertion 

[Executable  Assertion  Checker 
Executable  Assertion  Checking 

Assistance 

[see:  On-Line  Assistance] 

Attributes 

[Central  Role  of  Attributes  and  Functions 
Key  Attributes  of  Whole  APSEs 

Audit 

[Auditing 

Auditor 

Augmentability 

Autonomy 

Backup 

[see:  Archive,  Backup,  and  Retrieval  System] 
Benchmarks  and  Test  Suites 
Body  Stub  Generation 
Calculator 
Calendar 
Capacity 
Change 

[see  also:  Product:  Engineering  Change  Proposal, 
Specification  Change  Notice] 
[Change  Impact  Analysis 
Change  Request  Analysis 
Change  Request  Analyzer 
Change  Requirements 

CIDS 

[see:  Product:  Critical  Item  Development  Specification] 

Classification  Schema 

Coding  And  CSU  Testing 

Command  Language 

[Command  Language  Processing 
Command  Language  Processor 


5.1.2] 


5.3.4; 

7. 1.6.6] 


5.7.6; 
7.3. 2.6] 


(6.) 

2.3: 

3.3] 


7.3.1.22; 

5.6.21] 

(6.4.4) 

(6.4.5) 


(3.4.1) 
(7.1.6.11) 

(5.5.2) 
(5.5.7) 
(6.4.6) 


7.3.6. 1; 
7.3.6; 
5.2.9; 
4.11] 


(2.1) 

(4.6) 

7.2.3. 1; 
5.1.1] 
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Commonality  (Data  and  Communication) 

(6.4.7) 

Communication  Effectiveness 

(6.4.8) 

Communicativeness 

[Operability 

6.4.20] 

Comparator 

(5.6.1) 

Comparison 

(7.3.1. 1) 

Compile 

[Compilation 

7.1. 6.7; 

Compilation  System: 

5.3, 

Compiler 

5.3.3] 

Completeness 

(6.4.9) 

[Completeness  Checker 

5.6.11; 

Completeness  Checking 

7.3.1.12] 

Complexity 

[Complexity  Measurement 

7.3.1.10; 

Complexity  Measurer 

5.6.9] 

Component 

[APSE  Tool  Categories 

5.] 

Computer 

[see  also:  Product:  Computer  Resources  Integrated 

Support  Document] 

[Computer  Management  System 

5.1; 

Computer  System  Management 

7.2.3] 

Concept 

[System  Concepts 

4.1] 

Conferencing 

[see:  Electronic] 

Configuration 

[Configuration  Management 

7.2. 2. 7; 

Life  Cycle  Activities 

4.x. 5, 

Configuration  Manager 

5.2.7, 

Required  Configuration 

6.4.26] 

Consistency 

(6.4.10) 

[Consistency  Checker 

5.6.12; 

Consistency  Checking 

7.3.1.13] 

Constraint 

[Constraint  Evaluation 

7.3.2.7; 

Constraint  Evaluator 

5.7.7] 

Contention 

[see:  Constraint] 
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Conversion 

(7.1. 6.8) 

Correctness 

(6.2.1) 

[Correctness  Checker 

5.6.10: 

Correctness  Checking 

7.3.1.11] 

Cost 

(6.4.11) 

[Cost  Estimator 

5.2.1: 

Cost  Estimation 

7.2.2. 1] 

Coverage 

[Coverage/Frequency  Analysis 

7.3.2.8; 

Coverage/Frequency  Analyzer 

5.7.8] 

CRISD 

[see:  Product:  Computer  Resources  Integrated 

Support  Document] 

Cross  Reference 

[Cross  Reference 

7.3.1.17; 

Cross  Referencer 

5.6.16] 

CSC  Integration  And  Testing 

(4.7) 

CSCI  Testing 

(4.8) 

CSOM 

[see:  Product:  Computer  System  Operator's  Manual] 

CSU 

[Coding  And  CSU  Testing 

4.6] 

Data 

[Data  Base  Management 

7.2.1. 1; 

Data  Base  Manager 

5.1.9: 

Data  Editing 

7. 1.1. 2; 

Data  Row  Analysis 

7.3.1. 3; 

Data  Flow  Analyzer 

5.6.2: 

Test  Data  Management 

7.2.1. 8] 

Debug 

[Debugger 

5.7.5: 

Debugging 

7.3. 2. 5] 

Decision  Aids 

(3.4.3) 

Decompilation 

(7. 1.7.5) 

Design 

(6.2) 

[see  also:  Product: 


Interface  Design  Document, 
Software  Design  Document, 
System/Segment  Design  Document] 
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[Design  Attributes  (of  Whole  APSEs) 

Design  Generation 

Design  Prototype 

Design  Prototyping 

Detailed  Design 

Detailed  Design  Translation 

Preliminary  Design 

Preliminary  Design  Translation 

System  Requirements  Analysis/Design 

Desktop  System 

Detailed  Design 
[see:  Design] 

Development 

[see  also:  Methodology] 

[Life  Cycle  Activities:  System  Development  Management 

Dictionary 

Disassembling 

Distributedness 

Document 

[see  also:  Product] 

[Document  Accessibility 
Document  System 
Document  System 
Document  Manager 
Documentation  Management 

Dynamic 

[Dynamic  Analysis 
Dynamic  Analyzer  System 

E&V 

[Approaches  to  Whole-APSE  E&V 
E&V  Categories 

Locating  and  Applying  E&V  Technology 

ECP 

[see:  Product:  Engineering  Change  Proposal] 

Edit 

[Editing: 

Text 

Data 

Graphics 

Syntax-Directed  Editor 
Word  Processor 


3.3.2; 

7.1.7.1; 

5.7.4; 

7. 3. 2. 4; 
4.5: 

7.1.6.5: 

4.4; 

7. 1.6.4; 
4.2] 

(5.5) 


4.X.1] 
(5.5.8) 
(7.1. 7.6) 
(6.4.12) 


6.4.13: 

5.4, 

5.4, 
5.4.1; 
7.2.1. 2] 


7.3.2; 

5.7] 


3.4; 

2.4.2; 

2.4.2] 


7.1.1, 

7.1. 1.1, 

7.1. 1.2, 
7.1. 1.3: 
5.3.2; 
5.4.2] 
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Efficiency 

Electronic 

[APSE  Tool  Categories: 

Electronic  Conferencing 
Electronic  Mail 
Functions: 

Electronic  Conferencing 
Electronic  Mail 

Emulate 

[Emulation 

Emulator 

Engineering 

[Life  Cycle  Activities:  System  Engineering 
Environment 

[APSE  Definitions  and  Alternate  Names 
Error 

[Error  Checker 
Error  Checking 

Error  Tolerance  (Anomaly  Management) 

Estimate 

[Cost  Estimation 
Cost  Estimator 
Resource  Estimation 
Resource  Estimator 

Evaluation 

[see  also:  E&V] 

[Constraint  Evaluation 
Evaluation  Results  Management 
Life  Cycle  Activities:  System  Product  Evaluations 
Operational  Testing  And  Evaluation 

Execute 

[Executable  Assertion  Checker 
Executable  Assertion  Checking 
Processing  (Execution)  Effectiveness 
Symbolic  Execution 
Symbolic  Execution  System 

Expandability 

Expansion 

[Macro  Expansion 

Experiments,  Monitored 


(6.1.1) 


5.5.6, 

5.5.4; 

7. 2. 1.5, 
7.2.1. 4] 


7.3.2.13; 

5.7.13] 


4.X. 2] 


3.1] 


5.6.22; 

7.3.1.23; 

6.4.2] 


7.2.2.1; 

5.2.1; 

7.2.2.5; 

5.2.5] 


7.3.2.71 
7.2.1. 9; 
4.X.4; 
4.10] 


5.7.6; 

7.3.2.61 

6.4.22; 

7.3.4; 

5.7.19] 

(6.3.1) 


7.1. 6.9] 
(3.4.4) 
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Expert  System,  APSE  Viewed  as  a  Knowledge-Based 

Export 

[see:  Import/Export) 

Fault  Tolerance 

[Anomaly  Management 

File 

[File  Management 
File  Manager 

Flexibility 

[Expandability 

Forms 

[Predefined  and  User-Defined  Forms 

Formal 

[Formal  Verification 
Formal  Verification  System 
Life  Cycle  Activities:  Formal  Qualification  Testing 

Format 

[Formal  Verification  System 

[Formatter 

Formatting: 

MIL-STD  Format 
Table  of  Contents 

Framework,  APSE  Viewed  as  a  Stable 

Frequency 

[see:  Coverage] 

FSM 

[see:  Product:  Firmware  Support  Manual] 

Function 

[Functional  Analysis 
Functional  Analyzer 
Functional  Overlap 
Functional  Scope 
Functions 

Generality 

Global 

Granularity 

Graphics 

[Graphics  Editing 
Graphics  Generation 
Graphics  Generator 


(3.2.5) 


6.4.2] 


7.2.1. 3; 
5.1.6] 


6.3.1] 


7.1. 2.3] 


7.3.3: 
5.7.18; 
4.x. 3] 


5.7.18] 

5.4.5; 

7.1.2, 

7. 1.2.1; 
7.1. 2.2] 

(3.2.6) 


7.3.1. 4; 

5.6.3; 

6.4.14; 

6.4.15; 

7-] 

(6.4.16) 
(4.12) 

(6.4.17) 

7.1. 1.3; 

7.1.5; 

5.4.4] 
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Help 

[see:  On-Line  Assistance] 

I/O 

[I/O  Specification  Analysis 
I/O  Specification  Analyzer 
Inpui/Output  Support 

IDD 

[see:  Product:  Interface  Design  Document] 

Import/Export 

[Import/Export  System 

Input 

[see:  I/O] 

Index 

[Index  Relationships 
Using  The  Reference  Indexes  Directly 

Information  Management 

[APSE  Viewed  as  an  Information  Management  System 
Functions:  Information  Management 

Integration 

[CSC  Integration  And  Testing 
System  Integration  And  Testing 

Integrity 

Interactive  System,  APSE  Viewed  as  a  User-Oriented 
Interface 

[see  also:  Product:  Interface  Design  Document, 

Interface  Requirements  Specification] 

[Interface  Analysis 
Interface  Analyzer 

Interoperability 

Interpret 

[Interpretation 

Interpreter 

Invocation 

[Invocation  Analysis 
Invocation  Analyzer 

IPSE 

[see  also:  APSE] 

[Life  Cycle  Activities 

IRS 

[see:  Product:  Interface  Requirements  Specification] 


7.3.1.29; 

5.6.28: 

7. 2. 3. 2] 


(7. 2. 3. 6) 
5.1.7] 


2.4; 

2.3.1] 


3.2.3: 

7.2.1] 


4.7: 

4.9] 

(6.1.2) 

(3.2.4) 


7.3.1. 5: 
5.6.4] 

(6.3.2) 


7.1.6.14; 

5.3.7] 


7.3.1.19: 

5.6.18] 


4.] 
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Job  Scheduler 
Kernel 
Kernel 
Language 

(see  also:  Command  Language] 

[Requirements  To  Natural  Language  Translation 

Library 

[Program  Library  Management 
Program  Library  Manager 
Runtime  Library 

Life  Cycle  Activities 

Link 

[Linker 

Linking/Loading 

Load 

[Linking/Loading 

Loader 

Macro  Expansion 
Mail 

[see:  Electronic] 

Maintainability 

[Maintainability  Analysis 
Maintainability  Analyzer 

Management 

[APSE  Tool  Categories: 

Computer  Management  System 
Project  Management  System 
APSE  Viewed  as  an  Information  Management  System 
Attributes: 

Anomaly  Management 
Functions: 

Computer  System  Management 
Configuration  Management 
Data  Base  Management 
Documentation  Management 
Evaluation  Results  Management 
Evaluation  Results  Management 
File  Management 
Information  Management 
Program  Library  Management 


(5.1.4) 
(7.2. 3. 3) 
(7.2.3. 3) 


7.1. 6.3] 


7.2.1. 7: 

5.3.1: 

5.3.8] 

(4.) 


5.3.5: 

7.1.6.13] 


7.1.6.13; 

5.3.6] 

(7.1. 6.9) 


(6.2.2) 

7.3.1.18; 

5.6.17] 


5.1, 

5.2; 

3.2.3: 

6.4.2: 

7.2, 
7.2.3, 

7.2. 2.7, 

7.2.1. 1, 

7.2. 1.2, 
7.2.1. 9, 
7.2. 1.9, 

7.2.1. 3, 
7.2.1, 

7.2.1. 7, 
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Project  Management 
Specification  Management 
Test  Data  Management 

Life  Cycle  Activities:  Configuration  Management 

Life  Cycle  Activities:  Software  Development  Management 

Math/Statistics 

Maturity 

Merge 

[Sort/Merge 

Methodology-Support  System,  APSE  Viewed  as  a 
Modeling 

[Simulation  And  Modeling 
Simulation  And  Modeling  Tools 

Modularity 

Monitoring 

[Performance  Monitoring 
Mutation 

[Mutation  Analysis 
Mutation  Analyzer 

Natural  Language 

[Requirements  To  Natural  Language  Translation 

On-Line  Assistance 

[On-Line  Assistance  Processing 
On-Line  Assistance  Processor 

Operability 

Operational  Testing  And  Evaluation 

Output 

[see:  I/O] 

Performance 

[Performance  Attributes  (of  Whole  APSEs) 

Performance  Monitoring 

Phone  Book 

PIDS 

[see:  Product:  Prime  Item  Development  Specification] 
Power 

Preamble  Generation 
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7.2.2, 
7.2.1. 6, 
7.2.1.8: 
4.x. 5, 
4.X.1] 

(7.2. 3. 4) 

(6.4.18) 


7.1.4] 

(3.2.2) 


7.3. 2. 3: 
5.7.3] 

(6.4.19) 


7.2.1.10] 


7. 3. 2. 9; 
5.7.9] 


7.1. 6.3] 


7.1.3; 

5.1.8] 

(6.4.20) 

(4.10) 


(6.1) 

3.3.1; 

7.2.1.10] 

(5.5.5) 


(6.4.21) 

(7.1.6.12) 
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Preliminary  Design 

[see:  Design] 

Problem  Report 

(Problem  Report  Analysis 
Problem  Report  Analyzer 

Processing  (Execution)  Effectiveness 

Product 

[Computer  Resources  Integrated  Support  Document  (CRISD): 
Detailed  Design 
Preliminary  Design 
Software  Requirements  Analysis 
System  Requirements  Analysis/Design 
Computer  System  Operator’s  Manual  (CSOM): 

Coding  And  CSU  Testing 
CSC  Integration  And  Testing 
CSCI  Testing 
Detailed  Design 
Preliminary  Design 

Critical  Item  Development  Specification  (CIDS): 

System  Requirements  Analysis/Design 
Engineering  Change  Proposal  (ECP): 

Change  Requirements 
Firmware  Support  Manual  (FSM): 

Coding  And  CSU  Testing 
CSC  Integration  And  Testing 
CSCI  Testing 
Detailed  Design 

Interface  Design  Document  (IDD): 

CSCI  Testing 
Detailed  Design 
Preliminary  Design 

Interface  Requirements  Specification  (IRS): 

Software  Requirements  Analysis 
System  Requirements  Analysis/Design 
Life  Cycle  Activities:  System  Product  Evaluations 
Prime  Item  Development  Specification  (PIDS): 

System  Requirements  Analysis/Design 
Software  Design  Document  (SDD): 

CSCI  Testing 
Detailed  Design 
Preliminary  Design 
Software  Development  File  (SDF): 

Coding  And  CSU  Testing 
CSC  Integration  And  Testing 


7.3.5: 

5.2.8] 

(6.4.22) 


4.5.6, 

4.4.6, 

4.3.6, 
4.2.6; 

4.6.6, 

4.7.6, 

4.8.6, 

4.5.6, 
4.4.6: 

4.2.2: 

4.11.2; 

4.6.6, 

4.7.6, 

4.8.6, 
4.5.6: 

4.8.2, 

4.5.2, 
4.4.2: 

4.3.2, 
4.2.2; 
4.x. 4; 

4.2.2: 

4.8.2, 

4.5.2, 
4.4.2; 

4.6.2, 

4.7.2, 
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CSCI  Testing 
Detailed  Design 

Software  Development  Plan  (SDP): 

Coding  And  CSU  Testing 
CSC  Integration  And  Testing 
CSCI  Testing 
Detailed  Design 
Preliminary  Design 
Software  Requirements  Analysis 
System  Requirements  Analysis/Design 
Software  Product  Specification  (SPS): 

CSCI  Testing 

System  Integration  And  Testing 
Software  Programmer’s  Manual  (SPM): 
Coding  And  CSU  Testing 
CSC  Integration  And  Testing 
CSCI  Testing 
Detailed  Design 

Software  Requirements  Specification  (SRS): 
Software  Requirements  Analysis 
System  Requirements  Analysis/Design 
Software  Test  Description  (STD): 

Coding  And  CSU  Testing 
CSC  Integration  And  Testing 
CSCI  Testing 
Detailed  Design 
Software  Test  Plan  (STP): 

Preliminary  Design 
Software  Test  Report  (STR): 

CSCI  Testing 

Software  User’s  Manual  (SUM): 

Coding  And  CSU  Testing 
CSC  Integration  And  Testing 
CSCI  Testing 
Detailed  Design 
Preliminary  Design 
Specification  Change  Notice  (SCN): 

Change  Requirements 

System/Segment  Design  Document  (SSDD): 

System  Requirements  Analysis/Design 
System/Segment  Specification  (SSS): 

System  Requirements  Analysis/Design 
Version  Description  Document  (VDD): 

CSCI  Testing 

System  Integration  And  Testing 


4.8.2, 

4.5.2: 

4.6.1, 

4.7.1, 

4.8.1, 

4.5.1, 

4.4.1, 

4.3.1, 
4.2.1; 

4.8.2, 
4.9.2; 

4.6.6, 

4.7.6, 

4.8.6, 
4.5.6; 

4.3.2, 
4.2.2; 

4.6.3, 

4.7.3, 

4.8.3, 
4.5.3: 

4.4.3: 

4.8.3; 

4.6.6, 

4.7.6, 

4.8.6, 

4.5.6, 
4.4.6: 

4.11.5; 

4.2.2; 

4.2.2; 

4.8.5, 

4.9.5] 
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Program 

[see  also:  Product:  Software  Programmer’s  Manual] 
[Program  Generation 
Program  Library  Management 
Program  Library  Manager 

Project  Management 

[Project  Management  System 

Proprietary  Rights 

Prototype 

[Design  Prototype 
Design  Prototyping 
Requirements  Prototype 
Requirements  Prototyping 

Qualification 

[Life  Cycle  Activities:  Formal  Qualification  Testing 
Quality 

[Quality  Analysis 
Quality  Analyzer 
Quality  Analyzer 
Quality  Assessment 
Quality  Specification 

Questionnaires 

Reachability 

[Reachability  Analysis 
Reachability  Analyzer 

Real  Time 

[Real  Time  Analysis 
Real  Time  Analyzer 

Reconfigurability 

Reference 

[Cross  Reference 
Cross  Referencer 

Structure  And  Use  Of  The  Reference  Manual  2.] 

Regression  Testing 

[Regression  Testing  Analyzer 

Rehostability 

Reliability 

[Reliability  Analysis 
Reliability  Analyzer 


7.1. 7.3: 
7.2. 1.7; 
5.3.1] 

(7.2.2) 

5.2] 

(6.4.23) 


5.7.4; 

7. 3. 2. 4; 
5.7.2: 
7.3.2. 2] 


4.x. 3] 


7.3.1 .9: 

5.2.2; 

5.6.8: 

7. 2. 2. 8; 
7.2.2.2; 

(3.4.2) 


7.3.1.16: 

5.6.15] 


7.3.2.17; 

5.7.17] 

(6.4.24) 


7.3.1.17; 

5.6.16: 


(7.3.2.11) 

5.7.11] 

(6.4.25) 

(6.1.3) 

7.3.2.16: 

5.7.16] 


index-13 


E&V  Reference  Manual,  Version  1.1 


Requirements 

[see  also:  Product:  Interface  Requirements  Specification, 
Software  Requirements  Specification] 
[Change  Requirements 
Required  Configuration 
Requirements  Prototype 
Requirements  Prototyping 
Requirements  Reconstruction 
Requirements  Simulation 
Requirements  Simulator 

Requirements  To  Natural  Language  Translation 
Software  Requirements  Analysis 
Software  Requirements  Translation 
System  Requirements  Analysis/Design 
System  Requirements  Translation 

Resource 

[see  also:  Product:  Computer  Resources  Integrated 
Support  Document] 

[Resource  Controller 
Resource  Estimation 
Resource  Estimator 
Resource  Utilization 
Resource  Utilization  Analyzer 

Retargetability 

Retrieval 

[Archive,  Backup,  and  Retrieval  System 

Reusability 

[Reusability  Analysis 
Reusability  Analyzer 

Robustness 

[Anomaly  Management 
Runtime 

[Runtime  Library 
Runtime  Environment 

Scan 

[Scanner 

Scanning 

Schedule 

[Job  Scheduler 
Scheduler 
Scheduling 

SCN 

[see:  Product:  Specification  Change  Notice] 


4.11: 

6.4.26: 

5.7.2; 

7. 3. 2. 2; 

7.1.7.2: 

7.3.2.1: 

5.7.1: 

7.1. 6.3: 

4.3; 

7. 1.6.1; 
4.2; 

7.1. 6.2] 


5.1.5; 

7. 2. 2. 5; 
5.2.5; 
7.3.2.12; 
5.7.12] 

(6.4.27) 


5.1.2] 

(6.3.3) 

7.3.1.14; 

5.6.13] 


6.4.2] 


5.3.8; 
7.2.3. 5] 


5.6.19; 

7.3.1.20] 


5.1.4; 
5.2.3; 
7.2. 2. 3] 
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SDD 

[see:  Product:  Software  Design  Document] 

SDF 

[see:  Product:  Software  Development  File] 

SDP 

[see:  Product:  Software  Development  Plan] 

Security  System 
Self-Descriptiveness 
Simplicity 
Simulate 

[Requirements  Simulation 
Requirements  Simulator 
Simulation  and  Modeling 
Simulation  and  Modeling  Tools 

Sizing 

[Sizing  Analysis 
Sizing  Analyzer 

Software 

[Software-Oriented  Criteria: 

Software  Production  Vehicle(s) 

Software  Requirements: 

Software  Requirements  Analysis 
Software  Requirements  Translation 

Sort/Merge 

Source  Reconstruction 
Specification 

(see  also  :  Product:  Critical  Item  Development  Specification, 
Interface  Requirements  Specification, 
Prime  Item  Development  Specification, 
Software  Product  Specification, 
Software  Requirements  Specification, 
System/Segment  Specification] 

[I/O  Specification  Analysis 
I/O  Specification  Analyzer 
Quality  Specification 
Specification  Management 

Spelling 

[Spell  Checker 
Spelling  Checking 

SPM 

[see:  Product:  Software  Programmer’s  Manual] 


(5.1.3) 

(6.4.28) 

(6.3.29) 


7.3.2. 1; 
5.7.1; 
7.3. 2.3; 
5.7.3] 


7.3.1.30; 

5.6.29] 


6.4, 

6.4.30; 

4.3, 

7.1. 6.2] 

(7.1.4) 

(7. 1.7.4) 


7.3.1.29; 
5.6.28; 
7.2. 2.2; 
7.2.1. 6] 


5.4.3; 
7.3.1. 2] 
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Spreadsheet  (5.5.1) 

SPS 

[see:  Product:  Software  Product  Specification] 

SRS 

[see:  Product:  Software  Requirements  Specification] 

SSDD 

[see:  Product:  System/Segment  Design  Document] 

SSS 

[see:  Product:  System/Segment  Specification] 


Static 

[Static  Analysis  7.3.1; 

Static  Analyzer  System  5.6] 

Statistic 

[Math/Statistics  7.2.3. 4; 

Statistical  Analysis  7.1.3.24; 

Statistical  Analyzer  5.6.23; 

Statistical  Profiler  5.6.24; 

Statistical  Profiling  7.1.3.25] 

STD 

[see:  Product:  Software  Test  Description] 

Storage  Effectiveness  (6.4.31) 


STP 

[see:  Product:  Software  Test  Plan] 

STR 

[see:  Product:  Software  Test  Report] 


Structure 

[Structure  And  Use  Of  The  Reference  Manual  2.; 

Structure  Checker  5.6.25; 

Structure  Checking  7.3.1.26; 

Structure  Preprocessing  7.1.6.10] 


SUM 

[see:  Product:  Software  User’s  Manual] 

Support 

[see  also:  Product:  Computer  Resources  Integrated  Support  Document] 


[Life  Cycle  Activities:  Transitioning  To  System  Support  4.x.6] 

Survivability  (6.1.4) 

Symbolic  Execution  (7.3.4) 

[Symbolic  Execution  System  5.7.19] 
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Syntax 


[Syntax-Directed  Editor 

5.3.2; 

Syntax  And  Semantics  Checker 

5.6.14; 

Syntax  And  Semantics  Checking 

7.3.1.15] 

Synthesis 

(7.1.7) 

System 

[see  also:  Product: 

Computer  System  Operator’s  Manual, 
System/Segment  Design  Document, 

System/Segment  Specification] 

[Life  Cycle  Activities: 

System  Concepts 

4.1, 

System  Development  Management 

4.x. 1, 

System  Engineering 

4.x. 2, 

System  Integration  and  Testing 

4.9, 

System  Product  Evaluations 

4.x. 4, 

System  Requirements  Analysis/Design 

4.2. 

Transitioning  To  System  Support 

4.X.6; 

System  Accessibility 

6.4.32; 

System  Clarity 

6.4.33; 

System  Compatibility 

6.4.34; 

System  Requirements  Translation 

7.1 .6.1] 

Table  Of  Contents  Formatting 

(7.1. 2.2) 

Test 

[see  also:  Product:  Software  Test  Description, 

Software  Test  Plan, 

Software  Test  Procedure, 

Software  Test  Report] 

[Life  Cycle  Activities: 

Coding  and  CSU  Testing 

4.6, 

CSC  Integration  And  Testing 

4.7, 

CSCI  Testing 

4.8, 

Formal  Qualification  Testing 

4.x. 3, 

Operational  Testing  And  Evaluation 

4.10, 

System  Integration  And  Testing 

4.9; 

Regression  Testing 

7.3.2.11; 

Regression  Testing  Analyzer 

5.7.11; 

Test  Condition  Analysis 

7.3.1. 8; 

Test  Condition  Analyzer 

5.6.7; 

Test  Data  Management 

7.2.1. 8; 

Test  Suites,  Benchmarks  and 

3.4.1; 

Testability  (Verifiability) 

6.2.3; 

Testability  Analysis 

7.3.1. 7; 

Testability  Analyzer 

5.6.6; 
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Testing 

Testing  Analyzer 

Text  Editing 

Timing 

[Timing 

Timing  Analyzer 


Tool 

[APSE  Tool  Categories 
APSE  Viewed  as  a  Collection  of  Tool 


Traceability 

[Traceability  Analysis 
Traceability  Analyzer 

Tracking 

[Function 

Tool 


Training 


Transition  _  _  . 

[Life  Cycle  Activities:  Transitioning  To  System  Support 


Translation 


Transportability 

Tuning 

[Tuning  Analyzer 
Type 

[Type  Analysis 
Type  Analyzer 


[Coding  and  CSU  Testing 
Units  Analysis 
Units  Analyzer 


Usability 

Validation 
[see  E&V] 

VDD 

[see:  Product:  Version  Description  Document] 
Verifiability 


7.3.2.10: 

5.7.10] 

(7.1. 1.1) 


7.3.2.14; 

5.7.14] 


5.: 

3.2.1] 

(6.4.35) 
7.3.1. 6; 
5.6.5] 


7. 2. 2. 6: 
5.2.6] 

(6.4.36) 


4.x. 6] 

(7.1.6) 

(6.3.4) 

(7.3.2.15) 

5.7.15] 


7.3.1.27 

5.6.26] 


4.6: 

7.3.1.28: 

5.6.27] 

(6.1.5) 


(6.2.3) 
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Verification 


[Formal  Verification 

7.3.3; 

Formal  Verification 

5.7.18] 

Virtuality 

(6.4.37) 

Visibility 

(6.4.38) 

Walkthrough 

[Structured  Walkthrough 

7.3.1.21; 

Structured  Walkthrough  Tool 

5.6.20] 

Whole  APSE  Issues 

(3.) 

Word  Processor 

(5.4.2) 

Work  Breakdown  Structure 

[Function 

7.2. 2. 4; 

Tool 

5.2.4] 
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